Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uTo92-00ADz0-7n for pgsql-hackers@arkaria.postgresql.org; Mon, 23 Jun 2025 20:52:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uTo8z-006deW-7b for pgsql-hackers@arkaria.postgresql.org; Mon, 23 Jun 2025 20:51:57 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uTo8y-006deO-Se for pgsql-hackers@lists.postgresql.org; Mon, 23 Jun 2025 20:51:57 +0000 Received: from mout-p-102.mailbox.org ([2001:67c:2050:0:465::102]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1uTo8w-003hgo-2e for pgsql-hackers@lists.postgresql.org; Mon, 23 Jun 2025 20:51:57 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [10.196.197.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4bR0ck31nPz9sdj; Mon, 23 Jun 2025 22:51:50 +0200 (CEST) Date: Mon, 23 Jun 2025 22:51:49 +0200 From: Christoph Berg To: Tomas Vondra Cc: Andres Freund , Tomas Vondra , pgsql-hackers@lists.postgresql.org Subject: Re: pgsql: Introduce pg_shmem_allocations_numa view Message-ID: References: <6c9f9f7e-947b-4fc3-bdb6-b0696d7492e5@vondra.me> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Re: Tomas Vondra > Didn't you say the first ~35 addresses succeed, right? What about the > addresses after that? That was pg_shmem_allocations_numa. The pg_numa_query_pages() in there works (does not return -1), but then some of the status[] values are -14. When pg_buffercache_numa fails, pg_numa_query_pages() itself returns -14. The printed os_page_ptrs[] contents are the same for the failing and non-failing calls, so the problem is probably elsewhere. /* Fill pointers for all the memory pages. */ idx = 0; for (char *ptr = startptr; ptr < endptr; ptr += os_page_size) { + if (idx < 50) + elog(DEBUG1, "os_page_ptrs idx %d = %p", idx, ptr); os_page_ptrs[idx++] = ptr; 20:47 myon@postgres =# select * from pg_buffercache_numa; DEBUG: 00000: os_page_ptrs idx 0 = 0xebc44000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 1 = 0xebc45000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 2 = 0xebc46000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 3 = 0xebc47000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 4 = 0xebc48000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 5 = 0xebc49000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 6 = 0xebc4a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 7 = 0xebc4b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 8 = 0xebc4c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 9 = 0xebc4d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 10 = 0xebc4e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 11 = 0xebc4f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 12 = 0xebc50000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 13 = 0xebc51000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 14 = 0xebc52000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 15 = 0xebc53000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 16 = 0xebc54000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 17 = 0xebc55000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 18 = 0xebc56000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 19 = 0xebc57000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 20 = 0xebc58000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 21 = 0xebc59000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 22 = 0xebc5a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 23 = 0xebc5b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 24 = 0xebc5c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 25 = 0xebc5d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 26 = 0xebc5e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 27 = 0xebc5f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 28 = 0xebc60000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 29 = 0xebc61000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 30 = 0xebc62000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 31 = 0xebc63000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 32 = 0xebc64000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 33 = 0xebc65000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 34 = 0xebc66000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 35 = 0xebc67000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 36 = 0xebc68000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 37 = 0xebc69000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 38 = 0xebc6a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 39 = 0xebc6b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 40 = 0xebc6c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 41 = 0xebc6d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 42 = 0xebc6e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 43 = 0xebc6f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 44 = 0xebc70000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 45 = 0xebc71000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 46 = 0xebc72000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 47 = 0xebc73000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 48 = 0xebc74000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 49 = 0xebc75000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: NUMA: NBuffers=16384 os_page_count=32768 os_page_size=4096 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:385 2025-06-23 20:47:41.827 UTC [1368080] ERROR: failed NUMA pages inquiry: Bad address 2025-06-23 20:47:41.827 UTC [1368080] STATEMENT: select * from pg_buffercache_numa; ERROR: XX000: failed NUMA pages inquiry: Bad address LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:396 Time: 92.757 ms 20:47 myon@postgres =# select * from pg_buffercache_numa; DEBUG: 00000: os_page_ptrs idx 0 = 0xebc44000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 1 = 0xebc45000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 2 = 0xebc46000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 3 = 0xebc47000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 4 = 0xebc48000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 5 = 0xebc49000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 6 = 0xebc4a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 7 = 0xebc4b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 8 = 0xebc4c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 9 = 0xebc4d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 10 = 0xebc4e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 11 = 0xebc4f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 12 = 0xebc50000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 13 = 0xebc51000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 14 = 0xebc52000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 15 = 0xebc53000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 16 = 0xebc54000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 17 = 0xebc55000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 18 = 0xebc56000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 19 = 0xebc57000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 20 = 0xebc58000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 21 = 0xebc59000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 22 = 0xebc5a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 23 = 0xebc5b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 24 = 0xebc5c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 25 = 0xebc5d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 26 = 0xebc5e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 27 = 0xebc5f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 28 = 0xebc60000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 29 = 0xebc61000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 30 = 0xebc62000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 31 = 0xebc63000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 32 = 0xebc64000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 33 = 0xebc65000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 34 = 0xebc66000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 35 = 0xebc67000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 36 = 0xebc68000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 37 = 0xebc69000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 38 = 0xebc6a000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 39 = 0xebc6b000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 40 = 0xebc6c000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 41 = 0xebc6d000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 42 = 0xebc6e000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 43 = 0xebc6f000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 44 = 0xebc70000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 45 = 0xebc71000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 46 = 0xebc72000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 47 = 0xebc73000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 48 = 0xebc74000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: os_page_ptrs idx 49 = 0xebc75000 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:375 DEBUG: 00000: NUMA: NBuffers=16384 os_page_count=32768 os_page_size=4096 LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:385 DEBUG: 00000: NUMA: page-faulting the buffercache for proper NUMA readouts LOCATION: pg_buffercache_numa_pages, pg_buffercache_pages.c:444 Time: 24.547 ms 20:47 myon@postgres =# Christoph