Message-ID: <0643ae61-cf9d-482c-9b2c-fb861b24fd22@vondra.me>
Date: Mon, 23 Jun 2025 23:14:28 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pgsql: Introduce pg_shmem_allocations_numa view
To: Christoph Berg <myon@debian.org>
Cc: Andres Freund <andres@anarazel.de>,
 Tomas Vondra <tomas.vondra@postgresql.org>,
 pgsql-hackers@lists.postgresql.org
References: <aFln0lUZke2eEZsP@msg.df7cb.de> <aFlpO2lBsMjqarbp@msg.df7cb.de>
 <g3mywoeo7jmh6rci7epx2ishowgz65q2j7ek3c5f3lxcmvuktg@ler2fsv4szmn>
 <aFlwrNqKjHC1vj8h@msg.df7cb.de> <aFl53B_JNN3i-viX@msg.df7cb.de>
 <kl4zd72eeaex7zcicpuvpsuslrs5nfvmab7xzt4jnvcjvd6mxw@tcp64c55qkpj>
 <aFmxxHVS9XwbQ_em@msg.df7cb.de>
 <6c9f9f7e-947b-4fc3-bdb6-b0696d7492e5@vondra.me>
 <aFm5tmSFwhGX7mA7@msg.df7cb.de>
 <ce305c79-0a68-46ce-b563-ab9f87bb5f20@vondra.me>
 <aFm-ZfL-vz9I2Zmc@msg.df7cb.de>
Content-Language: en-US
From: Tomas Vondra <tomas@vondra.me>
In-Reply-To: <aFm-ZfL-vz9I2Zmc@msg.df7cb.de>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/0643ae61-cf9d-482c-9b2c-fb861b24fd22%40vondra.me>
Precedence: bulk


On 6/23/25 22:51, Christoph Berg wrote:
> Re: Tomas Vondra
>> Didn't you say the first ~35 addresses succeed, right? What about the
>> addresses after that?
> 
> That was pg_shmem_allocations_numa. The pg_numa_query_pages() in there
> works (does not return -1), but then some of the status[] values are
> -14.
> 
> When pg_buffercache_numa fails, pg_numa_query_pages() itself
> returns -14.
> 
> The printed os_page_ptrs[] contents are the same for the failing and
> non-failing calls, so the problem is probably elsewhere.
> 
>         /* Fill pointers for all the memory pages. */
>         idx = 0;
>         for (char *ptr = startptr; ptr < endptr; ptr += os_page_size)
>         {
> +           if (idx < 50)
> +               elog(DEBUG1, "os_page_ptrs idx %d = %p", idx, ptr);
>             os_page_ptrs[idx++] = ptr;
> 
> 
> 20:47 myon@postgres =# select * from pg_buffercache_numa;
> DEBUG:  00000: os_page_ptrs idx 0 = 0xebc44000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 1 = 0xebc45000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 2 = 0xebc46000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 3 = 0xebc47000
...
> DEBUG:  00000: os_page_ptrs idx 47 = 0xebc73000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 48 = 0xebc74000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 49 = 0xebc75000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: NUMA: NBuffers=16384 os_page_count=32768 os_page_size=4096
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:385
> 2025-06-23 20:47:41.827 UTC [1368080] ERROR:  failed NUMA pages inquiry: Bad address
> 2025-06-23 20:47:41.827 UTC [1368080] STATEMENT:  select * from pg_buffercache_numa;
> ERROR:  XX000: failed NUMA pages inquiry: Bad address
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:396
> Time: 92.757 ms
> 
> 20:47 myon@postgres =# select * from pg_buffercache_numa;
> DEBUG:  00000: os_page_ptrs idx 0 = 0xebc44000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 1 = 0xebc45000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 2 = 0xebc46000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 3 = 0xebc47000
...> DEBUG:  00000: os_page_ptrs idx 46 = 0xebc72000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 47 = 0xebc73000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 48 = 0xebc74000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: os_page_ptrs idx 49 = 0xebc75000
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:375
> DEBUG:  00000: NUMA: NBuffers=16384 os_page_count=32768 os_page_size=4096
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:385
> DEBUG:  00000: NUMA: page-faulting the buffercache for proper NUMA readouts
> LOCATION:  pg_buffercache_numa_pages, pg_buffercache_pages.c:444
> Time: 24.547 ms
> 20:47 myon@postgres =#
> 

True. If it fails on first call, but succeeds on the other, then the
problem is likely somewhere else. But also on the second call we won't
do the memory touching. Can you try setting firstNumaTouch=false, so
that we do this on every call?


At the beginning you mentioned this is happening on i386, armel and
armhf - are all those in qemu? I've tried on my rpi5 (with 32-bit user
space), and there everything seems to work fine. But that's aarch64
kernel, just the user space if 32-bit.


regards

-- 
Tomas Vondra