Date: Tue, 24 Jun 2025 08:24:53 +0000
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Tomas Vondra <tomas@vondra.me>
Cc: Christoph Berg <myon@debian.org>, Andres Freund <andres@anarazel.de>,
	Tomas Vondra <tomas.vondra@postgresql.org>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: pgsql: Introduce pg_shmem_allocations_numa view
Message-ID: <aFpg1de9ZfS1QgUt@ip-10-97-1-34.eu-west-3.compute.internal>
References: <kl4zd72eeaex7zcicpuvpsuslrs5nfvmab7xzt4jnvcjvd6mxw@tcp64c55qkpj>
 <aFmxxHVS9XwbQ_em@msg.df7cb.de>
 <6c9f9f7e-947b-4fc3-bdb6-b0696d7492e5@vondra.me>
 <aFm5tmSFwhGX7mA7@msg.df7cb.de>
 <ce305c79-0a68-46ce-b563-ab9f87bb5f20@vondra.me>
 <aFm-ZfL-vz9I2Zmc@msg.df7cb.de>
 <0643ae61-cf9d-482c-9b2c-fb861b24fd22@vondra.me>
 <aFnGQ2lfb8cfukim@msg.df7cb.de>
 <cd16c4a3-2860-46de-a74e-5532d1b2ee39@vondra.me>
 <6342f601-77de-4ee0-8c2a-3deb50ceac5b@vondra.me>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="YnYdx3wnSYItcXOD"
Content-Disposition: inline
In-Reply-To: <6342f601-77de-4ee0-8c2a-3deb50ceac5b@vondra.me>
Archived-At: 
 <https://www.postgresql.org/message-id/aFpg1de9ZfS1QgUt%40ip-10-97-1-34.eu-west-3.compute.internal>
Precedence: bulk


--YnYdx3wnSYItcXOD
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi,

On Tue, Jun 24, 2025 at 03:43:19AM +0200, Tomas Vondra wrote:
> On 6/23/25 23:47, Tomas Vondra wrote:
> > ...
> > 
> > Or maybe the 32-bit chroot on 64-bit host matters and confuses some
> > calculation.
> >
> 
> I think it's likely something like this.

I think the same.

> I noticed that if I modify
> pg_buffercache_numa_pages() to query the addresses one by one, it works.
> And when I increase the number, it stops working somewhere between 16k
> and 17k items.

Yeah, same for me with pg_get_shmem_allocations_numa(). It works if
pg_numa_query_pages() is done on chunks <= 16 pages but fails if done on more
than 16 pages.

It's also confirmed by test_chunk_size.c attached:

$ gcc-11 -m32 -o test_chunk_size test_chunk_size.c
$ ./test_chunk_size
 1 pages: SUCCESS (0 errors)
 2 pages: SUCCESS (0 errors)
 3 pages: SUCCESS (0 errors)
 4 pages: SUCCESS (0 errors)
 5 pages: SUCCESS (0 errors)
 6 pages: SUCCESS (0 errors)
 7 pages: SUCCESS (0 errors)
 8 pages: SUCCESS (0 errors)
 9 pages: SUCCESS (0 errors)
10 pages: SUCCESS (0 errors)
11 pages: SUCCESS (0 errors)
12 pages: SUCCESS (0 errors)
13 pages: SUCCESS (0 errors)
14 pages: SUCCESS (0 errors)
15 pages: SUCCESS (0 errors)
16 pages: SUCCESS (0 errors)
17 pages: 1 errors
Threshold: 17 pages

No error if -m32 is not used.

> It may be a coincidence, but I suspect it's related to the sizeof(void
> *) being 8 in the kernel, but only 4 in the chroot. So the userspace
> passes an array of 4-byte items, but kernel interprets that as 8-byte
> items. That is, we call
> 
> long move_pages(int pid, unsigned long count, void *pages[.count], const
> int nodes[.count], int status[.count], int flags);
> 
> Which (I assume) just passes the parameters to kernel. And it'll
> interpret them per kernel pointer size.
>

I also suspect something in this area...

> If this is what's happening, I'm not sure what to do about it ...

We could work by chunks (16?) on 32 bits but would probably produce performance
degradation (we mention it in the doc though). Also would always 16 be a correct
chunk size? 

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

--YnYdx3wnSYItcXOD
Content-Type: text/x-csrc; charset=us-ascii
Content-Disposition: attachment; filename="test_chunk_size.c"

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <string.h>
#include <errno.h>

int test_chunk_size(int chunk_size) {
    size_t page_size = sysconf(_SC_PAGESIZE);
    
    void *mem = malloc(page_size * chunk_size);
    if (!mem) return -1;
    
    memset(mem, 0xFF, page_size * chunk_size);
    
    void **ptrs = malloc(sizeof(void*) * chunk_size);
    int *status = malloc(sizeof(int) * chunk_size);
    
    for (int j = 0; j < chunk_size; j++) {
        ptrs[j] = (char*)mem + (j * page_size);
        status[j] = -999;
    }
    
    long result = syscall(SYS_move_pages, 0, chunk_size, ptrs, NULL, status, 0);
    
    int errors = 0;
    if (result == 0) {
        for (int j = 0; j < chunk_size; j++) {
            if (status[j] < 0) errors++;
        }
    }
    
    free(mem);
    free(ptrs);
    free(status);
    
    return (result == 0) ? errors : -1;
}

int main() {
    int threshold = -1;
    
    // Test sizes from 1 to 40 pages
    for (int size = 1; size <= 40; size++) {
        int errors = test_chunk_size(size);
        
        if (errors == -1) {
            if (threshold == -1) threshold = size;
            break;
        } else if (errors == 0) {
            printf("%2d pages: SUCCESS (0 errors)\n", size);
        } else {
            printf("%2d pages: %d errors\n", 
                   size, errors);
            threshold = size;
            break;
        }
    }
    
    if (threshold > 0)
        printf("Threshold: %d pages\n", threshold);
     else 
        printf("No threshold found in range 1-40 pages\n");
    
    return 0;
}

--YnYdx3wnSYItcXOD--