public inbox for [email protected]
help / color / mirror / Atom feedAdding basic NUMA awareness
89+ messages / 10 participants
[nested] [flat]
* Adding basic NUMA awareness
@ 2025-07-01 19:07 Tomas Vondra <[email protected]>
0 siblings, 4 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-07-01 19:07 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hi,
This is a WIP version of a patch series I'm working on, adding some
basic NUMA awareness for a couple parts of our shared memory (shared
buffers, etc.). It's based on Andres' experimental patches he spoke
about at pgconf.eu 2024 [1], and while it's improved and polished in
various ways, it's still experimental.
But there's a recent thread aiming to do something similar [2], so
better to share it now so that we can discuss both approaches. This
patch set is a bit more ambitious, handling NUMA in a way to allow
smarter optimizations later, so I'm posting it in a separate thread.
The series is split into patches addressing different parts of the
shared memory, starting (unsurprisingly) from shared buffers, then
buffer freelists and ProcArray. There's a couple additional parts, but
those are smaller / addressing miscellaneous stuff.
Each patch has a numa_ GUC, intended to enable/disable that part. This
is meant to make development easier, not as a final interface. I'm not
sure how exactly that should look. It's possible some combinations of
GUCs won't work, etc.
Each patch should have a commit message explaining the intent and
implementation, and then also detailed comments explaining various
challenges and open questions.
But let me go over the basics, and discuss some of the design choices
and open questions that need solving.
1) v1-0001-NUMA-interleaving-buffers.patch
This is the main thing when people think about NUMA - making sure the
shared buffers are allocated evenly on all the nodes, not just on a
single node (which can happen easily with warmup). The regular memory
interleaving would address this, but it also has some disadvantages.
Firstly, it's oblivious to the contents of the shared memory segment,
and we may not want to interleave everything. It's also oblivious to
alignment of the items (a buffer can easily end up "split" on multiple
NUMA nodes), or relationship between different parts (e.g. there's a
BufferBlock and a related BufferDescriptor, and those might again end up
on different nodes).
So the patch handles this by explicitly mapping chunks of shared buffers
to different nodes - a bit like interleaving, but in larger chunks.
Ideally each node gets (1/N) of shared buffers, as a contiguous chunk.
It's a bit more complicated, because the patch distributes both the
blocks and descriptors, in the same way. So a buffer and it's descriptor
always end on the same NUMA node. This is one of the reasons why we need
to map larger chunks, because NUMA works on page granularity, and the
descriptors are tiny - many fit on a memory page.
There's a secondary benefit of explicitly assigning buffers to nodes,
using this simple scheme - it allows quickly determining the node ID
given a buffer ID. This is helpful later, when building freelist.
The patch is fairly simple. Most of the complexity is about picking the
chunk size, and aligning the arrays (so that it nicely aligns with
memory pages).
The patch has a GUC "numa_buffers_interleave", with "off" by default.
2) v1-0002-NUMA-localalloc.patch
This simply sets "localalloc" when initializing a backend, so that all
memory allocated later is local, not interleaved. Initially this was
necessary because the patch set the allocation policy to interleaving
before initializing shared memory, and we didn't want to interleave the
private memory. But that's no longer the case - the explicit mapping to
nodes does not have this issue. I'm keeping the patch for convenience,
it allows experimenting with numactl etc.
The patch has a GUC "numa_localalloc", with "off" by default.
3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
Minor optimization. Andres noticed we're tracking the tail of buffer
freelist, without using it. So the patch removes that.
4) v1-0004-NUMA-partition-buffer-freelist.patch
Right now we have a single freelist, and in busy instances that can be
quite contended. What's worse, the freelist may trash between different
CPUs, NUMA nodes, etc. So the idea is to have multiple freelists on
subsets of buffers. The patch implements multiple strategies how the
list can be split (configured using "numa_partition_freelist" GUC), for
experimenting:
* node - One list per NUMA node. This is the most natural option,
because we now know which buffer is on which node, so we can ensure a
list for a node only has buffers from that list.
* cpu - One list per CPU. Pretty simple, each CPU gets it's own list.
* pid - Similar to "cpu", but the processes are mapped to lists based on
PID, not CPU ID.
* none - nothing, sigle freelist
Ultimately, I think we'll want to go with "node", simply because it
aligns with the buffer interleaving. But there are improvements needed.
The main challenge is that with multiple smaller lists, a process can't
really use the whole shared buffers. So a single backed will only use
part of the memory. The more lists there are, the worse this effect is.
This is also why I think we won't use the other partitioning options,
because there's going to be more CPUs than NUMA nodes.
Obviously, this needs solving even with NUMA nodes - we need to allow a
single backend to utilize the whole shared buffers if needed. There
should be a way to "steal" buffers from other freelists (if the
"regular" freelist is empty), but the patch does not implement this.
Shouldn't be hard, I think.
The other missing part is clocksweep - there's still just a single
instance of clocksweep, feeding buffers to all the freelists. But that's
clearly a problem, because the clocksweep returns buffers from all NUMA
nodes. The clocksweep really needs to be partitioned the same way as a
freelists, and each partition will operate on a subset of buffers (from
the right NUMA node).
I do have a separate experimental patch doing something like that, I
need to make it part of this branch.
5) v1-0005-NUMA-interleave-PGPROC-entries.patch
Another area that seems like it might benefit from NUMA is PGPROC, so I
gave it a try. It turned out somewhat challenging. Similarly to buffers
we have two pieces that need to be located in a coordinated way - PGPROC
entries and fast-path arrays. But we can't use the same approach as for
buffers/descriptors, because
(a) Neither of those pieces aligns with memory page size (PGPROC is
~900B, fast-path arrays are variable length).
(b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
rather high max_connections before we use multiple huge pages.
The fast-path arrays are less of a problem, because those tend to be
larger, and are accessed through pointers, so we can just adjust that.
So what I did instead is splitting the whole PGPROC array into one array
per NUMA node, and one array for auxiliary processes and 2PC xacts. So
with 4 NUMA nodes there are 5 separate arrays, for example. Each array
is a multiple of memory pages, so we may waste some of the memory. But
that's simply how NUMA works - page granularity.
This however makes one particular thing harder - in a couple places we
accessed PGPROC entries through PROC_HDR->allProcs, which was pretty
much just one large array. And GetNumberFromPGProc() relied on array
arithmetics to determine procnumber. With the array partitioned, this
can't work the same way.
But there's a simple solution - if we turn allProcs into an array of
*pointers* to PGPROC arrays, there's no issue. All the places need a
pointer anyway. And then we need an explicit procnumber field in PGPROC,
instead of calculating it.
There's a chance this have negative impact on code that accessed PGPROC
very often, but so far I haven't seen such cases. But if you can come up
with such examples, I'd like to see those.
There's another detail - when obtaining a PGPROC entry in InitProcess(),
we try to get an entry from the same NUMA node. And only if that doesn't
work, we grab the first one from the list (there's still just one PGPROC
freelist, I haven't split that - maybe we should?).
This has a GUC "numa_procs_interleave", again "off" by default. It's not
quite correct, though, because the partitioning happens always. It only
affects the PGPROC lookup. (In a way, this may be a bit broken.)
6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch
This is an experimental patch, that simply pins the new process to the
NUMA node obtained from the freelist.
Driven by GUC "numa_procs_pin" (default: off).
Summary
-------
So this is what I have at the moment. I've tried to organize the patches
in the order of importance, but that's just my guess. It's entirely
possible there's something I missed, some other order might make more
sense, etc.
There's also the question how this is related to other patches affecting
shared memory - I think the most relevant one is the "shared buffers
online resize" by Ashutosh, simply because it touches the shared memory.
I don't think the splitting would actually make some things simpler, or
maybe more flexible - in particular, it'd allow us to enable huge pages
only for some regions (like shared buffers), and keep the small pages
e.g. for PGPROC. So that'd be good.
But there'd also need to be some logic to "rework" how shared buffers
get mapped to NUMA nodes after resizing. It'd be silly to start with
memory on 4 nodes (25% each), resize shared buffers to 50% and end up
with memory only on 2 of the nodes (because the other 2 nodes were
originally assigned the upper half of shared buffers).
I don't have a clear idea how this would be done, but I guess it'd
require a bit of code invoked sometime after the resize. It'd already
need to rebuild the freelists in some way, I guess.
The other thing I haven't thought about very much is determining on
which CPUs/nodes the instance is allowed to run. I assume we'd start by
simply inherit/determine that at the start through libnuma, not through
some custom PG configuration (which the patch [2] proposed to do).
regards
[1] https://www.youtube.com/watch?v=V75KpACdl6E
[2]
https://www.postgresql.org/message-id/CAKZiRmw6i1W1AwXxa-Asrn8wrVcVH3TO715g_MCoowTS9rkGyw%40mail.gma...
--
Tomas Vondra
Attachments:
[text/x-patch] v1-0001-NUMA-interleaving-buffers.patch (26.9K, ../../[email protected]/2-v1-0001-NUMA-interleaving-buffers.patch)
download | inline diff:
From 9712e50d6d15c18ea2c5fcf457972486b0d4ef53 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 6 May 2025 21:12:21 +0200
Subject: [PATCH v1 1/6] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure not to
split a buffer on different NUMA nodes (which with the regular
interleaving is guaranteed to happen, unless when using huge pages). The
patch performs "explicit" interleaving, so that buffers are not split
like this.
The patch maps both buffers and buffer descriptors, so that the buffer
and it's buffer descriptor end up on the same NUMA node.
The mapping happens in larger chunks (see choose_chunk_items). This is
required to handle buffer descriptors (which are smaller than buffers),
and it should also help to reduce the number of mappings. Most NUMA
systems will use 1GB chunks, unless using very small shared buffers.
Notes:
* The feature is enabled by numa_buffers_interleave GUC (false by default)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
src/backend/storage/buffer/buf_init.c | 384 +++++++++++++++++++++++++-
src/backend/storage/buffer/bufmgr.c | 1 +
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/bin/pgbench/pgbench.c | 67 ++---
src/include/miscadmin.h | 2 +
src/include/storage/bufmgr.h | 1 +
7 files changed, 427 insertions(+), 41 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..2ad34624c49 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -25,6 +33,19 @@ WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+static Size get_memory_page_size(void);
+static int64 choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes);
+static void pg_numa_interleave_memory(char *startptr, char *endptr,
+ Size mem_page_size, Size chunk_size,
+ int num_nodes);
+
+/* number of buffers allocated on the same NUMA node */
+static int64 numa_chunk_buffers = -1;
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1;
+
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -71,18 +92,80 @@ BufferManagerShmemInit(void)
foundDescs,
foundIOCV,
foundBufCkpt;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +195,63 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available
+ * NUMA nodes. We can't use the regular interleaving, because with
+ * regular memory pages (smaller than BLCKSZ) we'd split all buffers
+ * to multiple NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have
+ * mapping for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the
+ * same NUMA node, as calculated by choose_chunk_buffers(). We try to
+ * keep the chunks large enough to work both for buffers and buffer
+ * descriptors, but not too large. See the comments at
+ * choose_chunk_buffers() for details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ */
+ if (numa_buffers_interleave)
+ {
+ char *startptr,
+ *endptr;
+ Size chunk_size;
+
+ numa_nodes = numa_num_configured_nodes();
+
+ numa_chunk_buffers
+ = choose_chunk_buffers(NBuffers, mem_page_size, numa_nodes);
+
+ elog(LOG, "BufferManagerShmemInit num_nodes %d chunk_buffers %ld",
+ numa_nodes, numa_chunk_buffers);
+
+ /* first map buffers */
+ startptr = BufferBlocks;
+ endptr = startptr + ((Size) NBuffers) * BLCKSZ;
+ chunk_size = (numa_chunk_buffers * BLCKSZ);
+
+ pg_numa_interleave_memory(startptr, endptr,
+ mem_page_size,
+ chunk_size,
+ numa_nodes);
+
+ /* now do the same for buffer descriptors */
+ startptr = (char *) BufferDescriptors;
+ endptr = startptr + ((Size) NBuffers) * sizeof(BufferDescPadded);
+ chunk_size = (numa_chunk_buffers * sizeof(BufferDescPadded));
+
+ pg_numa_interleave_memory(startptr, endptr,
+ mem_page_size,
+ chunk_size,
+ numa_nodes);
+ }
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +284,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +297,72 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ Size mem_page_size;
+
+ /* XXX why does IsUnderPostmaster matter? */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(mem_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(mem_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -186,3 +379,178 @@ BufferManagerShmemSize(void)
return size;
}
+
+/*
+ * choose_chunk_buffers
+ * choose the number of buffers allocated to a NUMA node at once
+ *
+ * We don't map shared buffers to NUMA nodes one by one, but in larger chunks.
+ * This is both for efficiency reasons (fewer mappings), and also because we
+ * want to map buffer descriptors too - and descriptors are much smaller. So
+ * we pick a number that's high enough for descriptors to use whole pages.
+ *
+ * We also want to keep buffers somehow evenly distributed on nodes, with
+ * about NBuffers/nodes per node. So we don't use chunks larger than this,
+ * to keep it as fair as possible (the chunk size is a possible difference
+ * between memory allocated to different NUMA nodes).
+ *
+ * It's possible shared buffers are so small this is not possible (i.e.
+ * it's less than chunk_size). But sensible NUMA systems will use a lot
+ * of memory, so this is unlikely.
+ *
+ * We simply print a warning about the misbalance, and that's it.
+ *
+ * XXX It'd be good to ensure the chunk size is a power-of-2, because then
+ * we could calculate the NUMA node simply by shift/modulo, while now we
+ * have to do a division. But we don't know how many buffers and buffer
+ * descriptors fits into a memory page. It may not be a power-of-2.
+ */
+static int64
+choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes)
+{
+ int64 num_items;
+ int64 max_items;
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(mem_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % mem_page_size) == 0) || ((mem_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of items to fill a memory page with descriptors and
+ * blocks. The NUMA allocates memory in pages, and we need to do that for
+ * both buffers and descriptors.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ * But it's clearer this way.
+ */
+ num_items = Max(mem_page_size / sizeof(BufferDescPadded),
+ mem_page_size / BLCKSZ);
+
+ /*
+ * We shouldn't use chunks larger than NBuffers/num_nodes, because with
+ * larger chunks the last NUMA node would end up with much less memory (or
+ * no memory at all).
+ */
+ max_items = (NBuffers / num_nodes);
+
+ /*
+ * Did we already exceed the maximum desirable chunk size? That is, will
+ * the last node get less than one whole chunk (or no memory at all)?
+ */
+ if (num_items > max_items)
+ elog(WARNING, "choose_chunk_buffers: chunk items exceeds max (%ld > %ld)",
+ num_items, max_items);
+
+ /* grow the chunk size until we hit the max limit. */
+ while (2 * num_items <= max_items)
+ num_items *= 2;
+
+ /*
+ * XXX It's not difficult to construct cases where we end up with not
+ * quite balanced distribution. For example, with shared_buffers=10GB and
+ * 4 NUMA nodes, we end up with 2GB chunks, which means the first node
+ * gets 4GB, and the three other nodes get 2GB each.
+ *
+ * We could be smarter, and try to get more balanced distribution. We
+ * could simply reduce max_items e.g. to
+ *
+ * max_items = (NBuffers / num_nodes) / 4;
+ *
+ * in which cases we'd end up with 512MB chunks, and each nodes would get
+ * the same 2.5GB chunk. It may not always work out this nicely, but it's
+ * better than with (NBuffers / num_nodes).
+ *
+ * Alternatively, we could "backtrack" - try with the large max_items,
+ * check how balanced it is, and if it's too imbalanced, try with a
+ * smaller one.
+ *
+ * We however want a simple scheme.
+ */
+
+ return num_items;
+}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_chunk_buffers == -1)
+ return -1;
+
+ return (buffer / numa_chunk_buffers) % numa_nodes;
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_interleave_memory(char *startptr, char *endptr,
+ Size mem_page_size, Size chunk_size,
+ int num_nodes)
+{
+ volatile uint64 touch pg_attribute_unused();
+ char *ptr = startptr;
+
+ /* chunk size has to be a multiple of memory page */
+ Assert((chunk_size % mem_page_size) == 0);
+
+ /*
+ * Walk the memory pages in the range, and determine the node for each
+ * one. We use numa_tonode_memory(), because then we can move a whole
+ * memory range to the node, we don't need to worry about individual pages
+ * like with numa_move_pages().
+ */
+ while (ptr < endptr)
+ {
+ /* We may have an incomplete chunk at the end. */
+ Size sz = Min(chunk_size, (endptr - ptr));
+
+ /*
+ * What NUMA node does this range belong to? Each chunk should go to
+ * the same NUMA node, in a round-robin manner.
+ */
+ int node = ((ptr - startptr) / chunk_size) % num_nodes;
+
+ /*
+ * Nope, we have the first buffer from the next memory page, and we'll
+ * set NUMA node for it (and all pages up to the next buffer). The
+ * buffer should align with the memory page, thanks to the
+ * buffer_align earlier.
+ */
+ Assert((int64) ptr % mem_page_size == 0);
+ Assert((sz % mem_page_size) == 0);
+
+ /*
+ * XXX no return value, to make this fail on error, has to use
+ * numa_set_strict
+ *
+ * XXX Should we still touch the memory first, like with numa_move_pages,
+ * or is that not necessary?
+ */
+ numa_tonode_memory(ptr, sz, node);
+
+ ptr += sz;
+ }
+
+ /* should have processed all chunks */
+ Assert(ptr == endptr);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 406ce77693c..e1e1cfd379d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -685,6 +685,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
BufferDesc *bufHdr;
BufferTag tag;
uint32 buf_state;
+
Assert(BufferIsValid(recent_buffer));
ResourceOwnerEnlarge(CurrentResourceOwner);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 511dc32d519..198a57e70a5 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 69b6a877dc9..c07de903f76 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -305,7 +305,7 @@ static const char *progname;
#define CPU_PINNING_RANDOM 1
#define CPU_PINNING_COLOCATED 2
-static int pinning_mode = CPU_PINNING_NONE;
+static int pinning_mode = CPU_PINNING_NONE;
#define WSEP '@' /* weight separator */
@@ -874,20 +874,20 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
*/
typedef struct cpu_generator_state
{
- int ncpus; /* number of CPUs available */
- int nitems; /* number of items in the queue */
- int *nthreads; /* number of threads for each CPU */
- int *nclients; /* number of processes for each CPU */
- int *items; /* queue of CPUs to pick from */
-} cpu_generator_state;
+ int ncpus; /* number of CPUs available */
+ int nitems; /* number of items in the queue */
+ int *nthreads; /* number of threads for each CPU */
+ int *nclients; /* number of processes for each CPU */
+ int *items; /* queue of CPUs to pick from */
+} cpu_generator_state;
static cpu_generator_state cpu_generator_init(int ncpus);
-static void cpu_generator_refill(cpu_generator_state *state);
-static void cpu_generator_reset(cpu_generator_state *state);
-static int cpu_generator_thread(cpu_generator_state *state);
-static int cpu_generator_client(cpu_generator_state *state, int thread_cpu);
-static void cpu_generator_print(cpu_generator_state *state);
-static bool cpu_generator_check(cpu_generator_state *state);
+static void cpu_generator_refill(cpu_generator_state * state);
+static void cpu_generator_reset(cpu_generator_state * state);
+static int cpu_generator_thread(cpu_generator_state * state);
+static int cpu_generator_client(cpu_generator_state * state, int thread_cpu);
+static void cpu_generator_print(cpu_generator_state * state);
+static bool cpu_generator_check(cpu_generator_state * state);
static void reset_pinning(TState *threads, int nthreads);
@@ -7422,7 +7422,7 @@ main(int argc, char **argv)
/* try to assign threads/clients to CPUs */
if (pinning_mode != CPU_PINNING_NONE)
{
- int nprocs = get_nprocs();
+ int nprocs = get_nprocs();
cpu_generator_state state = cpu_generator_init(nprocs);
retry:
@@ -7433,6 +7433,7 @@ retry:
for (i = 0; i < nthreads; i++)
{
TState *thread = &threads[i];
+
thread->cpu = cpu_generator_thread(&state);
}
@@ -7444,7 +7445,7 @@ retry:
while (true)
{
/* did we find any unassigned backend? */
- bool found = false;
+ bool found = false;
for (i = 0; i < nthreads; i++)
{
@@ -7678,10 +7679,10 @@ threadRun(void *arg)
/* determine PID of the backend, pin it to the same CPU */
for (int i = 0; i < nstate; i++)
{
- char *pid_str;
- pid_t pid;
+ char *pid_str;
+ pid_t pid;
- PGresult *res = PQexec(state[i].con, "select pg_backend_pid()");
+ PGresult *res = PQexec(state[i].con, "select pg_backend_pid()");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
pg_fatal("could not determine PID of the backend for client %d",
@@ -8184,7 +8185,7 @@ cpu_generator_init(int ncpus)
{
struct timeval tv;
- cpu_generator_state state;
+ cpu_generator_state state;
state.ncpus = ncpus;
@@ -8207,7 +8208,7 @@ cpu_generator_init(int ncpus)
}
static void
-cpu_generator_refill(cpu_generator_state *state)
+cpu_generator_refill(cpu_generator_state * state)
{
struct timeval tv;
@@ -8223,7 +8224,7 @@ cpu_generator_refill(cpu_generator_state *state)
}
static void
-cpu_generator_reset(cpu_generator_state *state)
+cpu_generator_reset(cpu_generator_state * state)
{
state->nitems = 0;
cpu_generator_refill(state);
@@ -8236,15 +8237,15 @@ cpu_generator_reset(cpu_generator_state *state)
}
static int
-cpu_generator_thread(cpu_generator_state *state)
+cpu_generator_thread(cpu_generator_state * state)
{
if (state->nitems == 0)
cpu_generator_refill(state);
while (true)
{
- int idx = lrand48() % state->nitems;
- int cpu = state->items[idx];
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
state->items[idx] = state->items[state->nitems - 1];
state->nitems--;
@@ -8256,10 +8257,10 @@ cpu_generator_thread(cpu_generator_state *state)
}
static int
-cpu_generator_client(cpu_generator_state *state, int thread_cpu)
+cpu_generator_client(cpu_generator_state * state, int thread_cpu)
{
- int min_clients;
- bool has_valid_cpus = false;
+ int min_clients;
+ bool has_valid_cpus = false;
for (int i = 0; i < state->nitems; i++)
{
@@ -8284,8 +8285,8 @@ cpu_generator_client(cpu_generator_state *state, int thread_cpu)
while (true)
{
- int idx = lrand48() % state->nitems;
- int cpu = state->items[idx];
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
if (cpu == thread_cpu)
continue;
@@ -8303,7 +8304,7 @@ cpu_generator_client(cpu_generator_state *state, int thread_cpu)
}
static void
-cpu_generator_print(cpu_generator_state *state)
+cpu_generator_print(cpu_generator_state * state)
{
for (int i = 0; i < state->ncpus; i++)
{
@@ -8312,10 +8313,10 @@ cpu_generator_print(cpu_generator_state *state)
}
static bool
-cpu_generator_check(cpu_generator_state *state)
+cpu_generator_check(cpu_generator_state * state)
{
- int min_count = INT_MAX,
- max_count = 0;
+ int min_count = INT_MAX,
+ max_count = 0;
for (int i = 0; i < state->ncpus; i++)
{
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..c257c8a1c20 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -319,6 +319,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
--
2.49.0
[text/x-patch] v1-0002-NUMA-localalloc.patch (3.7K, ../../[email protected]/3-v1-0002-NUMA-localalloc.patch)
download | inline diff:
From 6919b1c1c59a6084017ebae5a884bb6c60639364 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v1 2/6] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 16 ++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 28 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 43b4dbccc3d..d11936691b2 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,18 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 198a57e70a5..57f2df7ab74 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.49.0
[text/x-patch] v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../../[email protected]/4-v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From c2b2edb71d629ebe4283b636f058b8e42d1f1a35 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v1 3/6] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.49.0
[text/x-patch] v1-0004-NUMA-partition-buffer-freelist.patch (19.0K, ../../[email protected]/5-v1-0004-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From 6505848ac8359c8c76dfbffc7150b6601ab07601 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v1 4/6] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
There are four strategies, specified by GUC numa_partition_freelist
* none - single long freelist, should work just like now
* node - one freelist per NUMA node, with only buffers from that node
* cpu - one freelist per CPU
* pid - freelist determined by PID (same number of freelists as 'cpu')
When allocating a buffer, it's taken from the correct freelist (e.g.
same NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
src/backend/storage/buffer/buf_init.c | 4 +-
src/backend/storage/buffer/freelist.c | 324 +++++++++++++++++++++++---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 18 ++
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 8 +
6 files changed, 327 insertions(+), 29 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 2ad34624c49..920f1a32a8f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -543,8 +543,8 @@ pg_numa_interleave_memory(char *startptr, char *endptr,
* XXX no return value, to make this fail on error, has to use
* numa_set_strict
*
- * XXX Should we still touch the memory first, like with numa_move_pages,
- * or is that not necessary?
+ * XXX Should we still touch the memory first, like with
+ * numa_move_pages, or is that not necessary?
*/
numa_tonode_memory(ptr, sz, node);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..c93ec2841c5 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,41 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
/*
* The shared freelist control information.
@@ -39,8 +66,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,13 +76,27 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
static BufferStrategyControl *StrategyControl = NULL;
+/*
+ * XXX shouldn't this be in BufferStrategyControl? Probably not, we need to
+ * calculate it during sizing, and perhaps it could change before the memory
+ * gets allocated (so we need to remember the values).
+ */
+static int strategy_nnodes;
+static int strategy_ncpus;
+
/*
* Private (non-shared) state for managing a ring of shared buffers to re-use.
* This is currently the only kind of BufferAccessStrategy object, but someday
@@ -157,6 +196,90 @@ ClockSweepTick(void)
return victim;
}
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ int freelist_idx;
+
+ /* freelist not partitioned, return the first (and only) freelist */
+ if (numa_partition_freelist == FREELIST_PARTITION_NONE)
+ return &StrategyControl->freelists[0];
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * FIXME This doesn't work well if CPUs are excluded from being run or
+ * offline. In that case we end up not using some freelists at all, but
+ * not sure if we need to worry about that. Probably not for now. But
+ * could that change while the system is running?
+ *
+ * XXX Maybe we should somehow detect changes to the list of CPUs, and
+ * rebuild the lists if that changes? But that seems expensive.
+ */
+ if (cpu > strategy_ncpus)
+ elog(ERROR, "cpu out of range: %d > %u", cpu, strategy_ncpus);
+ else if (node > strategy_nnodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, strategy_nnodes);
+
+ /*
+ * Pick the freelist, based on CPU, NUMA node or process PID. This matches
+ * how we built the freelists above.
+ *
+ * XXX Can we rely on some of the values (especially strategy_nnodes) to
+ * be a power-of-2? Then we could replace the modulo with a mask, which is
+ * likely more efficient.
+ */
+ switch (numa_partition_freelist)
+ {
+ case FREELIST_PARTITION_CPU:
+ freelist_idx = cpu % strategy_ncpus;
+ break;
+
+ case FREELIST_PARTITION_NODE:
+ freelist_idx = node % strategy_nnodes;
+ break;
+
+ case FREELIST_PARTITION_PID:
+ freelist_idx = MyProcPid % strategy_ncpus;
+ break;
+
+ default:
+ elog(ERROR, "unknown freelist partitioning value");
+ }
+
+ return &StrategyControl->freelists[freelist_idx];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +291,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < strategy_ncpus; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +319,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +386,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +436,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -352,11 +493,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* StrategyFreeBuffer: put a buffer on the freelist
+ *
+ * XXX This calls ChooseFreeList() again, and it might return the freelist to
+ * a different freelist than it was taken from (either by a different backend,
+ * or perhaps even the same backend running on a different CPU). Is that good?
+ * Maybe we should try to balance this somehow, e.g. by choosing a random list,
+ * the shortest one, or something like that? But that breaks the whole idea of
+ * having freelists with buffers from a particular NUMA node.
*/
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ freelist = ChooseFreeList();
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +516,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +584,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int node = 0; node < strategy_ncpus; node++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[node];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "freelist %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ node,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -446,11 +634,33 @@ StrategyShmemSize(void)
{
Size size = 0;
+ /* FIXME */
+#ifdef USE_LIBNUMA
+ strategy_ncpus = numa_num_task_cpus();
+ strategy_nnodes = numa_num_task_nodes();
+#else
+ strategy_ncpus = 1;
+ strategy_nnodes = 1;
+#endif
+
+ Assert(strategy_nnodes <= strategy_ncpus);
+
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /*
+ * Allocate one frelist per CPU. We might use per-node freelists, but the
+ * assumption is the number of CPUs is less than number of NUMA nodes.
+ *
+ * FIXME This assumes the we have more CPUs than NUMA nodes, which seems
+ * like a safe assumption. But maybe we should calculate how many elements
+ * we actually need, depending on the GUC? Not a huge amount of memory.
+ */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ strategy_ncpus)));
return size;
}
@@ -466,6 +676,7 @@ void
StrategyInitialize(bool init)
{
bool found;
+ int buffers_per_cpu;
/*
* Initialize the shared buffer lookup hashtable.
@@ -484,23 +695,27 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ offsetof(BufferStrategyControl, freelists) +
+ (sizeof(BufferStrategyFreelist) * strategy_ncpus),
&found);
if (!found)
{
+ /*
+ * XXX Calling get_nprocs() may not be quite correct, because some of
+ * the processors may get disabled, etc.
+ */
+ int num_cpus = get_nprocs();
+
/*
* Only done once, usually in postmaster
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +726,61 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < strategy_ncpus; nfreelist++)
+ {
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+ }
+
+ /* buffers per CPU (also used for PID partitioning) */
+ buffers_per_cpu = (NBuffers / strategy_ncpus);
+
+ elog(LOG, "NBuffers: %d, nodes %d, ncpus: %d, divide: %d, remain: %d",
+ NBuffers, strategy_nnodes, strategy_ncpus,
+ buffers_per_cpu, NBuffers - (strategy_ncpus * buffers_per_cpu));
+
+ /*
+ * Walk through the buffers, add them to the correct list. Walk from
+ * the end, because we're adding the buffers to the beginning.
+ */
+ for (int i = NBuffers - 1; i >= 0; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+ BufferStrategyFreelist *freelist;
+ int belongs_to = 0; /* first freelist by default */
+
+ /*
+ * Split the freelist into partitions, if needed (or just keep the
+ * freelist we already built in BufferManagerShmemInit().
+ */
+ if ((numa_partition_freelist == FREELIST_PARTITION_CPU) ||
+ (numa_partition_freelist == FREELIST_PARTITION_PID))
+ {
+ belongs_to = (i % num_cpus);
+ }
+ else if (numa_partition_freelist == FREELIST_PARTITION_NODE)
+ {
+ /* determine NUMA node for buffer */
+ belongs_to = BufferGetNode(i);
+ }
+
+ /* add to the right freelist */
+ freelist = &StrategyControl->freelists[belongs_to];
+
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
}
else
Assert(!init);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..7febf3001a3 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+int numa_partition_freelist = 0;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 57f2df7ab74..e2361c161e6 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -491,6 +491,14 @@ static const struct config_enum_entry file_copy_method_options[] = {
{NULL, 0, false}
};
+static const struct config_enum_entry freelist_partition_options[] = {
+ {"none", FREELIST_PARTITION_NONE, false},
+ {"node", FREELIST_PARTITION_NODE, false},
+ {"cpu", FREELIST_PARTITION_CPU, false},
+ {"pid", FREELIST_PARTITION_PID, false},
+ {NULL, 0, false}
+};
+
/*
* Options for enum values stored in other modules
*/
@@ -5284,6 +5292,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_USERSET, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ FREELIST_PARTITION_NONE, freelist_partition_options,
+ NULL, NULL, NULL
+ },
+
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..17528439f07 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT int numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index c257c8a1c20..efb7e28c10f 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -93,6 +93,14 @@ typedef enum ExtendBufferedFlags
EB_LOCK_TARGET = (1 << 5),
} ExtendBufferedFlags;
+typedef enum FreelistPartitionMode
+{
+ FREELIST_PARTITION_NONE,
+ FREELIST_PARTITION_NODE,
+ FREELIST_PARTITION_CPU,
+ FREELIST_PARTITION_PID,
+} FreelistPartitionMode;
+
/*
* Some functions identify relations either by relation or smgr +
* relpersistence. Used via the BMR_REL()/BMR_SMGR() macros below. This
--
2.49.0
[text/x-patch] v1-0005-NUMA-interleave-PGPROC-entries.patch (34.9K, ../../[email protected]/6-v1-0005-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From 05c594ed8eb8a266a74038c3131d12bb03d897e3 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v1 5/6] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 62 ++---
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 368 +++++++++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 11 +-
11 files changed, 407 insertions(+), 62 deletions(-)
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index 48f10bec91e..90ddff37bc6 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -576,7 +576,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -635,7 +635,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 7e622ae4bd2..75c0e4bf53c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 0fec4f1f871..0044ef54363 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index c93ec2841c5..4e390a77a71 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -360,7 +360,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index e5b945a9ee3..3277480fbcf 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1650,7 +1650,7 @@ TransactionIdIsActive(TransactionId xid)
for (i = 0; i < arrayP->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
TransactionId pxid;
/* Fetch xid just once - see GetNewTransactionId */
@@ -1792,7 +1792,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2276,7 +2276,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2350,7 +2350,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2551,7 +2551,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2777,7 +2777,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2808,7 +2808,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -3058,7 +3058,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3099,7 +3099,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3227,7 +3227,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3270,7 +3270,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3339,7 +3339,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3441,7 +3441,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3506,7 +3506,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3561,7 +3561,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3607,7 +3607,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3636,7 +3636,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3667,7 +3667,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3708,7 +3708,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3771,7 +3771,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3837,7 +3837,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 2776ceb295b..95b1da42408 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..9d3e94a7b3a 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -89,6 +97,12 @@ static void ProcKill(int code, Datum arg);
static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* NUMA */
+static Size get_memory_page_size(void); /* XXX duplicate */
+static void move_to_node(char *startptr, char *endptr,
+ Size mem_page_size, int node);
+static int numa_nodes = -1;
+
/*
* Report shared-memory space needed by PGPROC.
@@ -100,11 +114,40 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * With NUMA, we allocate the PGPROC array in several chunks. With shared
+ * buffers we simply manually assign parts of the buffer array to
+ * different NUMA nodes, and that does the trick. But we can't do that for
+ * PGPROC, as the number of PGPROC entries is much lower, especially with
+ * huge pages. We can fit ~2k entries on a 2MB page, and NUMA does stuff
+ * with page granularity, and the large NUMA systems are likely to use
+ * huge pages. So with sensible max_connections we would not use more than
+ * a single page, which means it gets to a single NUMA node.
+ *
+ * So we allocate PGPROC not as a single array, but one array per NUMA
+ * node, and then one array for aux processes (without NUMA node
+ * assigned). Each array may need up to memory-page-worth of padding,
+ * worst case. So we just add that - it's a bit wasteful, but good enough
+ * for PoC.
+ *
+ * FIXME Should be conditional, but that was causing problems in bootstrap
+ * mode. Or maybe it was because the code that allocates stuff later does
+ * not do that conditionally. Anyway, needs to be fixed.
+ */
+ /* if (numa_procs_interleave) */
+ {
+ int num_nodes = numa_num_configured_nodes();
+ Size mem_page_size = get_memory_page_size();
+
+ size = add_size(size, mul_size((num_nodes + 1), mem_page_size));
+ }
+
return size;
}
@@ -129,6 +172,26 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * Same NUMA-padding logic as in PGProcShmemSize, adding a memory page per
+ * NUMA node - but this way we add two pages per node - one for PGPROC,
+ * one for fast-path arrays. In theory we could make this work just one
+ * page per node, by adding fast-path arrays right after PGPROC entries on
+ * each node. But now we allocate fast-path locks separately - good enough
+ * for PoC.
+ *
+ * FIXME Should be conditional, but that was causing problems in bootstrap
+ * mode. Or maybe it was because the code that allocates stuff later does
+ * not do that conditionally. Anyway, needs to be fixed.
+ */
+ /* if (numa_procs_interleave) */
+ {
+ int num_nodes = numa_num_configured_nodes();
+ Size mem_page_size = get_memory_page_size();
+
+ size = add_size(size, mul_size((num_nodes + 1), mem_page_size));
+ }
+
return size;
}
@@ -191,11 +254,13 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+ int procs_total;
+ int procs_per_node;
/* Used for setup of per-backend fast-path slots. */
char *fpPtr,
@@ -205,6 +270,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -224,6 +291,9 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* one chunk per NUMA node (without NUMA assume 1 node) */
+ numa_nodes = numa_num_configured_nodes();
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,19 +311,108 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it now.
+ */
+ procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+ procs_total = 0;
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ PGPROC *procs_node;
+
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int count_node = Min(procs_per_node, MaxBackends - procs_total);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(mem_page_size, ptr);
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+ ptr = (char *) ptr + count_node * sizeof(PGPROC);
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (j = 0; j < count_node; j++)
+ {
+ procs_node[j].numa_node = i;
+ procs_node[j].procnumber = procs_total;
+
+ ProcGlobal->allProcs[procs_total++] = &procs_node[j];
+ }
+
+ move_to_node((char *) procs_node, ptr, mem_page_size, i);
+ }
+
+ /*
+ * also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * don't assign those to any NUMA node)
+ *
+ * XXX Mostly duplicate of preceding block, could be reused.
+ */
+ {
+ PGPROC *procs_node;
+ int count_node = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /*
+ * Make sure to align PGPROC array to memory page (it may not be
+ * aligned). We won't assign this to any NUMA node, but we still don't
+ * want it to interfere with the preceding chunk (for the last NUMA
+ * node).
+ */
+ ptr = (char *) TYPEALIGN(mem_page_size, ptr);
+
+ procs_node = (PGPROC *) ptr;
+ ptr = (char *) ptr + count_node * sizeof(PGPROC);
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ /* now add the PGPROC pointers to allProcs */
+ for (j = 0; j < count_node; j++)
+ {
+ procs_node[j].numa_node = -1;
+ procs_node[j].procnumber = procs_total;
+
+ ProcGlobal->allProcs[procs_total++] = &procs_node[j];
+ }
+ }
+
+ /* we should have allocated the expected number of PGPROC entries */
+ Assert(procs_total == TotalProcs);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,23 +445,100 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /* reset the count */
+ procs_total = 0;
+
+ /*
+ * Mimic the same logic as above, but for fast-path locking.
+ */
+ for (i = 0; i < numa_nodes; i++)
{
- PGPROC *proc = &procs[i];
+ char *startptr;
+ char *endptr;
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int procs_node = Min(procs_per_node, MaxBackends - procs_total);
+
+ /* align to memory page, to make move_pages possible */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
+
+ startptr = fpPtr;
+ endptr = fpPtr + procs_node * (fpLockBitsSize + fpRelIdSize);
+
+ move_to_node(startptr, endptr, mem_page_size, i);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Now point the PGPROC entries to the fast-path arrays, and also
+ * advance the fpPtr.
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ for (j = 0; j < procs_node; j++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[procs_total++];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == i);
+ Assert(proc->procnumber == (procs_total - 1));
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We
+ * interleave the two arrays, to (hopefully) get some locality for
+ * each backend.
+ */
+ proc->fpLockBits = (uint64 *) fpPtr;
+ fpPtr += fpLockBitsSize;
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ proc->fpRelId = (Oid *) fpPtr;
+ fpPtr += fpRelIdSize;
- Assert(fpPtr <= fpEndPtr);
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(fpPtr == endptr);
+ }
+
+ /* auxiliary processes / prepared xacts */
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int procs_node = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* align to memory page, to make move_pages possible */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
+
+ /* now point the PGPROC entries to the fast-path arrays */
+ for (j = 0; j < procs_node; j++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[procs_total++];
+
+ /* cross-check we got PGPROC with no NUMA node assigned */
+ Assert(proc->numa_node == -1);
+ Assert(proc->procnumber == (procs_total - 1));
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We
+ * interleave the two arrays, to (hopefully) get some locality for
+ * each backend.
+ */
+ proc->fpLockBits = (uint64 *) fpPtr;
+ fpPtr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) fpPtr;
+ fpPtr += fpRelIdSize;
+
+ Assert(fpPtr <= fpEndPtr);
+ }
+ }
+
+ /* Should have consumed exactly the expected amount of fast-path memory. */
+ Assert(fpPtr <= fpEndPtr);
+
+ /* make sure we allocated the expected number of PGPROC entries */
+ Assert(procs_total == TotalProcs);
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -366,15 +602,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +668,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2259,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2334,60 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * move_to_node
+ * move all pages in the given range to the requested NUMA node
+ *
+ * XXX This is expected to only process fairly small number of pages, so no
+ * need to do batching etc. Just move pages one by one.
+ */
+static void
+move_to_node(char *startptr, char *endptr, Size mem_page_size, int node)
+{
+ while (startptr < endptr)
+ {
+ int r,
+ status;
+
+ r = numa_move_pages(0, 1, (void **) &startptr, &node, &status, 0);
+
+ if (r != 0)
+ elog(WARNING, "failed to move page to NUMA node %d (r = %d, status = %d)",
+ node, r, status);
+
+ startptr += mem_page_size;
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 7febf3001a3..bf775c76545 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
int numa_partition_freelist = 0;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e2361c161e6..930082588f2 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2144,6 +2144,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 17528439f07..f454b4e9d75 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9f9b3fcfbf1..5cb1632718e 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -194,6 +194,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -319,6 +321,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -383,7 +388,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -435,8 +440,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
--
2.49.0
[text/x-patch] v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch (3.4K, ../../[email protected]/7-v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From f76377a56f37421c61c4dd876813b57084b019df Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v1 6/6] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 9d3e94a7b3a..4c9e55608b2 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -729,6 +729,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index bf775c76545..e584ba840ef 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
int numa_partition_freelist = 0;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 930082588f2..3fc8897ae36 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2154,6 +2154,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f454b4e9d75..d0d960caa9d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.49.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-02 11:37 Ashutosh Bapat <[email protected]>
parent: Tomas Vondra <[email protected]>
3 siblings, 2 replies; 89+ messages in thread
From: Ashutosh Bapat @ 2025-07-02 11:37 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <[email protected]> wrote:
>
>
> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
>
> Minor optimization. Andres noticed we're tracking the tail of buffer
> freelist, without using it. So the patch removes that.
>
The patches for resizing buffers use the lastFreeBuffer to add new
buffers to the end of free list when expanding it. But we could as
well add it at the beginning of the free list.
This patch seems almost independent of the rest of the patches. Do you
need it in the rest of the patches? I understand that those patches
don't need to worry about maintaining lastFreeBuffer after this patch.
Is there any other effect?
If we are going to do this, let's do it earlier so that buffer
resizing patches can be adjusted.
>
> There's also the question how this is related to other patches affecting
> shared memory - I think the most relevant one is the "shared buffers
> online resize" by Ashutosh, simply because it touches the shared memory.
I have added Dmitry to this thread since he has written most of the
shared memory handling code.
>
> I don't think the splitting would actually make some things simpler, or
> maybe more flexible - in particular, it'd allow us to enable huge pages
> only for some regions (like shared buffers), and keep the small pages
> e.g. for PGPROC. So that'd be good.
The resizing patches split the shared buffer related structures into
separate memory segments. I think that itself will help enabling huge
pages for some regions. Would that help in your case?
>
> But there'd also need to be some logic to "rework" how shared buffers
> get mapped to NUMA nodes after resizing. It'd be silly to start with
> memory on 4 nodes (25% each), resize shared buffers to 50% and end up
> with memory only on 2 of the nodes (because the other 2 nodes were
> originally assigned the upper half of shared buffers).
>
> I don't have a clear idea how this would be done, but I guess it'd
> require a bit of code invoked sometime after the resize. It'd already
> need to rebuild the freelists in some way, I guess.
Yes, there's code to build the free list. I think we will need code to
remap the buffers and buffer descriptor.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-02 12:36 Tomas Vondra <[email protected]>
parent: Ashutosh Bapat <[email protected]>
1 sibling, 2 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-07-02 12:36 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On 7/2/25 13:37, Ashutosh Bapat wrote:
> On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <[email protected]> wrote:
>>
>>
>> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
>>
>> Minor optimization. Andres noticed we're tracking the tail of buffer
>> freelist, without using it. So the patch removes that.
>>
>
> The patches for resizing buffers use the lastFreeBuffer to add new
> buffers to the end of free list when expanding it. But we could as
> well add it at the beginning of the free list.
>
> This patch seems almost independent of the rest of the patches. Do you
> need it in the rest of the patches? I understand that those patches
> don't need to worry about maintaining lastFreeBuffer after this patch.
> Is there any other effect?
>
> If we are going to do this, let's do it earlier so that buffer
> resizing patches can be adjusted.
>
My patches don't particularly rely on this bit, it would work even with
lastFreeBuffer. I believe Andres simply noticed the current code does
not use lastFreeBuffer, it just maintains is, so he removed that as an
optimization. I don't know how significant is the improvement, but if
it's measurable we could just do that independently of our patches.
>>
>> There's also the question how this is related to other patches affecting
>> shared memory - I think the most relevant one is the "shared buffers
>> online resize" by Ashutosh, simply because it touches the shared memory.
>
> I have added Dmitry to this thread since he has written most of the
> shared memory handling code.
>
Thanks.
>>
>> I don't think the splitting would actually make some things simpler, or
>> maybe more flexible - in particular, it'd allow us to enable huge pages
>> only for some regions (like shared buffers), and keep the small pages
>> e.g. for PGPROC. So that'd be good.
>
> The resizing patches split the shared buffer related structures into
> separate memory segments. I think that itself will help enabling huge
> pages for some regions. Would that help in your case?
>
Indirectly. My patch can work just fine with a single segment, but being
able to enable huge pages only for some of the segments seems better.
>>
>> But there'd also need to be some logic to "rework" how shared buffers
>> get mapped to NUMA nodes after resizing. It'd be silly to start with
>> memory on 4 nodes (25% each), resize shared buffers to 50% and end up
>> with memory only on 2 of the nodes (because the other 2 nodes were
>> originally assigned the upper half of shared buffers).
>>
>> I don't have a clear idea how this would be done, but I guess it'd
>> require a bit of code invoked sometime after the resize. It'd already
>> need to rebuild the freelists in some way, I guess.
>
> Yes, there's code to build the free list. I think we will need code to
> remap the buffers and buffer descriptor.
>
Right. The good thing is that's just "advisory" information, it doesn't
break anything if it's temporarily out of sync. We don't need to "stop"
everything to remap the buffers to other nodes, or anything like that.
Or at least I think so.
It's one thing to "flip" the target mapping (determining which node a
buffer should be on), and actually migrating the buffers. The first part
can be done instantaneously, the second part can happen in the
background over a longer time period.
I'm not sure how you're rebuilding the freelist. Presumably it can
contain buffers that are no longer valid (after shrinking). How is that
handled to not break anything? I think the NUMA variant would do exactly
the same thing, except that there's multiple lists.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-03 14:07 Ashutosh Bapat <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Ashutosh Bapat @ 2025-07-03 14:07 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On Wed, Jul 2, 2025 at 6:06 PM Tomas Vondra <[email protected]> wrote:
>
> I'm not sure how you're rebuilding the freelist. Presumably it can
> contain buffers that are no longer valid (after shrinking). How is that
> handled to not break anything? I think the NUMA variant would do exactly
> the same thing, except that there's multiple lists.
Before shrinking the buffers, we walk the free list removing any
buffers that are going to be removed. When expanding, by linking the
new buffers in the order and then adding those to the already existing
free list. 0005 patch in [1] has the code for the same.
[1] https://www.postgresql.org/message-id/my4hukmejato53ef465ev7lk3sqiqvneh7436rz64wmtc7rbfj%40hmuxsf2ng...
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-03 14:49 Dmitry Dolgov <[email protected]>
parent: Ashutosh Bapat <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Dmitry Dolgov @ 2025-07-03 14:49 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
> On Wed, Jul 02, 2025 at 05:07:28PM +0530, Ashutosh Bapat wrote:
> > There's also the question how this is related to other patches affecting
> > shared memory - I think the most relevant one is the "shared buffers
> > online resize" by Ashutosh, simply because it touches the shared memory.
>
> I have added Dmitry to this thread since he has written most of the
> shared memory handling code.
Thanks! I like the idea behind this patch series. I haven't read it in
details yet, but I can imagine both patches (interleaving and online
resizing) could benefit from each other. In online resizing we've
introduced a possibility to use multiple shared mappings for different
types of data, maybe it would be convenient to use the same interface to
create separate mappings for different NUMA nodes as well. Using a
separate shared mapping per NUMA node would also make resizing easier,
since it would be more straightforward to fit an increased segment into
NUMA boundaries.
> > I don't think the splitting would actually make some things simpler, or
> > maybe more flexible - in particular, it'd allow us to enable huge pages
> > only for some regions (like shared buffers), and keep the small pages
> > e.g. for PGPROC. So that'd be good.
>
> The resizing patches split the shared buffer related structures into
> separate memory segments. I think that itself will help enabling huge
> pages for some regions. Would that help in your case?
Right, separate segments would allow to mix and match huge pages with
pages of regular size. It's not implemented in the latest version of
online resizing patch, purely to reduce complexity and maintain the same
invariant (everything is either using huge pages or not) -- but we could
do it other way around as well.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-04 11:05 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
3 siblings, 2 replies; 89+ messages in thread
From: Jakub Wartak @ 2025-07-04 11:05 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <[email protected]> wrote:
Hi!
> 1) v1-0001-NUMA-interleaving-buffers.patch
[..]
> It's a bit more complicated, because the patch distributes both the
> blocks and descriptors, in the same way. So a buffer and it's descriptor
> always end on the same NUMA node. This is one of the reasons why we need
> to map larger chunks, because NUMA works on page granularity, and the
> descriptors are tiny - many fit on a memory page.
Oh, now I get it! OK, let's stick to this one.
> I don't think the splitting would actually make some things simpler, or
> maybe more flexible - in particular, it'd allow us to enable huge pages
> only for some regions (like shared buffers), and keep the small pages
> e.g. for PGPROC. So that'd be good.
You have made assumption that this is good, but small pages (4KB) are
not hugetlb, and are *swappable* (Transparent HP are swappable too,
manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
most frequent problem I see these days are OOMs, and it makes me
believe that making certain critical parts of shared memory being
swappable just to make pagesize granular is possibly throwing the baby
out with the bathwater. I'm thinking about bad situations like: some
wrong settings of vm.swapiness that people keep (or distros keep?) and
general inability of PG to restrain from allocating more memory in
some cases.
> The other thing I haven't thought about very much is determining on
> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> simply inherit/determine that at the start through libnuma, not through
> some custom PG configuration (which the patch [2] proposed to do).
0. I think that we could do better, some counter arguments to
no-configuration-at-all:
a. as Robert & Bertrand already put it there after review: let's say I
want just to run on NUMA #2 node, so here I would need to override
systemd's script ExecStart= to include that numactl (not elegant?). I
could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
less friendly. Also it probably requires root to edit/reload systemd,
while having GUC for this like in my proposal makes it more smooth (I
think?)
b. wouldn't it be better if that stayed as drop-in rather than always
on? What if there's a problem, how do you disable those internal
optimizations if they do harm in some cases? (or let's say I want to
play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
numa_buffers_interleave would be nice?
c. What if I want my standby (walreceiver+startup/recovery) to run
with NUMA affinity to get better performance (I'm not going to hack
around systemd script every time, but I could imagine changing
numa=X,Y,Z after restart/before promotion)
d. Now if I would be forced for some reason to do that numactl(1)
voodoo, and use the those above mentioned overrides and PG wouldn't be
having GUC (let's say I would use `numactl
--weighted-interleave=0,1`), then:
> 2) v1-0002-NUMA-localalloc.patch
> This simply sets "localalloc" when initializing a backend, so that all
> memory allocated later is local, not interleaved. Initially this was
> necessary because the patch set the allocation policy to interleaving
> before initializing shared memory, and we didn't want to interleave the
> private memory. But that's no longer the case - the explicit mapping to
> nodes does not have this issue. I'm keeping the patch for convenience,
> it allows experimenting with numactl etc.
.. .is not accurate anymore and we would require to have that in
(still with GUC) ?
Thoughts? I can add that mine part into Your's patches if you want.
Way too quick review and some very fast benchmark probes, I've
concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
would be too new topic for me), but let's start:
1. normal pgbench -S (still with just s_b@4GB), done many tries,
consistent benefit for the patch with like +8..10% boost on generic
run:
numa_buffers_interleave=off numa_pgproc_interleave=on(due that
always on "if"), s_b just on 1 NUMA node (might happen)
latency average = 0.373 ms
latency stddev = 0.237 ms
initial connection time = 45.899 ms
tps = 160242.147877 (without initial connection time)
numa_buffers_interleave=on numa_pgproc_interleave=on
latency average = 0.345 ms
latency stddev = 0.373 ms
initial connection time = 44.485 ms
tps = 177564.686094 (without initial connection time)
2. Tested it the same way as I did for mine(problem#2 from Andres's
presentation): 4s32c128t, s_b=4GB (on 128GB), prewarm test (with
seqconcurrscans.pgb as earlier)
default/numa_buffers_interleave=off
latency average = 1375.478 ms
latency stddev = 1141.423 ms
initial connection time = 46.104 ms
tps = 45.868075 (without initial connection time)
numa_buffers_interleave=on
latency average = 838.128 ms
latency stddev = 498.787 ms
initial connection time = 43.437 ms
tps = 75.413894 (without initial connection time)
and i've repeated the the same test (identical conditions) with my
patch, got me slightly more juice:
latency average = 727.717 ms
latency stddev = 410.767 ms
initial connection time = 45.119 ms
tps = 86.844161 (without initial connection time)
(but mine didn't get that boost from normal pgbench as per #1
pgbench -S -- my numa='all' stays @ 160k TPS just as
numa_buffers_interleave=off), so this idea is clearly better.
So should I close https://commitfest.postgresql.org/patch/5703/
and you'll open a new one or should I just edit the #5703 and alter it
and add this thread too?
3. Patch is not calling interleave on PQ shmem, do we want to add that
in as some next item like v1-0007? Question is whether OS interleaving
makes sense there ? I believe it does there, please see my thread
(NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
being spawned by postmaster and may end up on different NUMA nodes
randomly, so actually OS-interleaving that memory reduces jitter there
(AKA bandwidth-over-latency). My thinking is that one cannot expect
static/forced CPU-to-just-one-NUMA-node assignment for backend and
it's PQ workers, because it is impossible have always available CPU
power there in that NUMA node, so it might be useful to interleave
that shared mem there too (as separate patch item?)
4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
(also in v1-0005). My worry is should we may put some
known-limitations docs (?) from start and mention that
if the VM is greatly resized and NUMA numa nodes appear, they might
not be used until restart?
5. In v1-0001, pg_numa_interleave_memory()
+ * XXX no return value, to make this fail on error, has to use
+ * numa_set_strict
Yes, my patch has those numa_error() and numa_warn() handlers too in
pg_numa. Feel free to use it for better UX.
+ * XXX Should we still touch the memory first, like
with numa_move_pages,
+ * or is that not necessary?
It's not necessary to touch after numa_tonode_memory() (wrapper around
numa_interleave_memory()), if it is going to be used anyway it will be
correctly placed to best of my knowledge.
6. diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
Accidental indents (also fails to apply)
7. We miss the pg_numa_* shims, but for sure that's for later and also
avoid those Linux specific #ifdef USE_LIBNUMA and so on?
8. v1-0005 2x + /* if (numa_procs_interleave) */
Ha! it's a TRAP! I've uncommented it because I wanted to try it out
without it (just by setting GUC off) , but "MyProc->sema" is NULL :
2025-07-04 12:31:08.103 CEST [28754] LOG: starting PostgreSQL
19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
[..]
2025-07-04 12:31:08.109 CEST [28754] LOG: io worker (PID 28755)
was terminated by signal 11: Segmentation fault
2025-07-04 12:31:08.109 CEST [28754] LOG: terminating any other
active server processes
2025-07-04 12:31:08.114 CEST [28754] LOG: shutting down because
"restart_after_crash" is off
2025-07-04 12:31:08.116 CEST [28754] LOG: database system is shut down
[New LWP 28755]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: io worker '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
at ./nptl/sem_waitcommon.c:136
136 ./nptl/sem_waitcommon.c: No such file or directory.
(gdb) where
#0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
at ./nptl/sem_waitcommon.c:136
#1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
#2 0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
../src/backend/port/posix_sema.c:302
#3 0x0000556191970553 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:992
#4 0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
../src/backend/postmaster/auxprocess.c:65
#5 0x0000556191940676 in IoWorkerMain (startup_data=<optimized
out>, startup_data_len=<optimized out>) at
../src/backend/storage/aio/method_worker.c:393
#6 0x00005561918e8163 in postmaster_child_launch
(child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
startup_data=startup_data@entry=0x0,
startup_data_len=startup_data_len@entry=0,
client_sock=client_sock@entry=0x0) at
../src/backend/postmaster/launch_backend.c:290
#7 0x00005561918ea09a in StartChildProcess
(type=type@entry=B_IO_WORKER) at
../src/backend/postmaster/postmaster.c:3973
#8 0x00005561918ea308 in maybe_adjust_io_workers () at
../src/backend/postmaster/postmaster.c:4404
[..]
(gdb) print *MyProc->sem
Cannot access memory at address 0x0
9. v1-0006: is this just a thought or serious candidate? I can imagine
it can easily blow-up with some backends somehow requesting CPUs only
from one NUMA node, while the second node being idle. Isn't it better
just to leave CPU scheduling, well, to the CPU scheduler? The problem
is that you have tools showing overall CPU usage, even mpstat(1) per
CPU , but no tools for per-NUMA node CPU util%, so it would be hard
for someone to realize that this is happening.
-J.
[1] - https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-04 18:12 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
1 sibling, 2 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-07-04 18:12 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/4/25 13:05, Jakub Wartak wrote:
> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <[email protected]> wrote:
>
> Hi!
>
>> 1) v1-0001-NUMA-interleaving-buffers.patch
> [..]
>> It's a bit more complicated, because the patch distributes both the
>> blocks and descriptors, in the same way. So a buffer and it's descriptor
>> always end on the same NUMA node. This is one of the reasons why we need
>> to map larger chunks, because NUMA works on page granularity, and the
>> descriptors are tiny - many fit on a memory page.
>
> Oh, now I get it! OK, let's stick to this one.
>
>> I don't think the splitting would actually make some things simpler, or
>> maybe more flexible - in particular, it'd allow us to enable huge pages
>> only for some regions (like shared buffers), and keep the small pages
>> e.g. for PGPROC. So that'd be good.
>
> You have made assumption that this is good, but small pages (4KB) are
> not hugetlb, and are *swappable* (Transparent HP are swappable too,
> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> most frequent problem I see these days are OOMs, and it makes me
> believe that making certain critical parts of shared memory being
> swappable just to make pagesize granular is possibly throwing the baby
> out with the bathwater. I'm thinking about bad situations like: some
> wrong settings of vm.swapiness that people keep (or distros keep?) and
> general inability of PG to restrain from allocating more memory in
> some cases.
>
I haven't observed such issues myself, or maybe I didn't realize it's
happening. Maybe it happens, but it'd be good to see some data showing
that, or a reproducer of some sort. But let's say it's real.
I don't think we should use huge pages merely to ensure something is not
swapped out. The "not swappable" is more of a limitation of huge pages,
not an advantage. You can't just choose to make them swappable.
Wouldn't it be better to keep using 4KB pages, but lock the memory using
mlock/mlockall?
>> The other thing I haven't thought about very much is determining on
>> which CPUs/nodes the instance is allowed to run. I assume we'd start by
>> simply inherit/determine that at the start through libnuma, not through
>> some custom PG configuration (which the patch [2] proposed to do).
>
> 0. I think that we could do better, some counter arguments to
> no-configuration-at-all:
>
> a. as Robert & Bertrand already put it there after review: let's say I
> want just to run on NUMA #2 node, so here I would need to override
> systemd's script ExecStart= to include that numactl (not elegant?). I
> could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
> less friendly. Also it probably requires root to edit/reload systemd,
> while having GUC for this like in my proposal makes it more smooth (I
> think?)
>
> b. wouldn't it be better if that stayed as drop-in rather than always
> on? What if there's a problem, how do you disable those internal
> optimizations if they do harm in some cases? (or let's say I want to
> play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
> numa_buffers_interleave would be nice?
>
> c. What if I want my standby (walreceiver+startup/recovery) to run
> with NUMA affinity to get better performance (I'm not going to hack
> around systemd script every time, but I could imagine changing
> numa=X,Y,Z after restart/before promotion)
>
> d. Now if I would be forced for some reason to do that numactl(1)
> voodoo, and use the those above mentioned overrides and PG wouldn't be
> having GUC (let's say I would use `numactl
> --weighted-interleave=0,1`), then:
>
I'm not against doing something like this, but I don't plan to do that
in V1. I don't have a clear idea what configurability is actually
needed, so it's likely I'd do the interface wrong.
>> 2) v1-0002-NUMA-localalloc.patch
>> This simply sets "localalloc" when initializing a backend, so that all
>> memory allocated later is local, not interleaved. Initially this was
>> necessary because the patch set the allocation policy to interleaving
>> before initializing shared memory, and we didn't want to interleave the
>> private memory. But that's no longer the case - the explicit mapping to
>> nodes does not have this issue. I'm keeping the patch for convenience,
>> it allows experimenting with numactl etc.
>
> .. .is not accurate anymore and we would require to have that in
> (still with GUC) ?
> Thoughts? I can add that mine part into Your's patches if you want.
>
I'm sorry, I don't understand what's the question :-(
> Way too quick review and some very fast benchmark probes, I've
> concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
> would be too new topic for me), but let's start:
>
> 1. normal pgbench -S (still with just s_b@4GB), done many tries,
> consistent benefit for the patch with like +8..10% boost on generic
> run:
>
> numa_buffers_interleave=off numa_pgproc_interleave=on(due that
> always on "if"), s_b just on 1 NUMA node (might happen)
> latency average = 0.373 ms
> latency stddev = 0.237 ms
> initial connection time = 45.899 ms
> tps = 160242.147877 (without initial connection time)
>
> numa_buffers_interleave=on numa_pgproc_interleave=on
> latency average = 0.345 ms
> latency stddev = 0.373 ms
> initial connection time = 44.485 ms
> tps = 177564.686094 (without initial connection time)
>
> 2. Tested it the same way as I did for mine(problem#2 from Andres's
> presentation): 4s32c128t, s_b=4GB (on 128GB), prewarm test (with
> seqconcurrscans.pgb as earlier)
> default/numa_buffers_interleave=off
> latency average = 1375.478 ms
> latency stddev = 1141.423 ms
> initial connection time = 46.104 ms
> tps = 45.868075 (without initial connection time)
>
> numa_buffers_interleave=on
> latency average = 838.128 ms
> latency stddev = 498.787 ms
> initial connection time = 43.437 ms
> tps = 75.413894 (without initial connection time)
>
> and i've repeated the the same test (identical conditions) with my
> patch, got me slightly more juice:
> latency average = 727.717 ms
> latency stddev = 410.767 ms
> initial connection time = 45.119 ms
> tps = 86.844161 (without initial connection time)
>
> (but mine didn't get that boost from normal pgbench as per #1
> pgbench -S -- my numa='all' stays @ 160k TPS just as
> numa_buffers_interleave=off), so this idea is clearly better.
Good, thanks for the testing. I should have done something like this
when I posted my patches, but I forgot about that (and the email felt
too long anyway).
But this actually brings an interesting question. What exactly should we
expect / demand from these patches? In my mind it'd primarily about
predictability and stability of results.
For example, the results should not depend on how was the database
warmed up - was it done by a single backend or many backends? Was it
restarted, or what? I could probably warmup the system very carefully to
ensure it's balanced. The patches mean I don't need to be that careful.
> So should I close https://commitfest.postgresql.org/patch/5703/
> and you'll open a new one or should I just edit the #5703 and alter it
> and add this thread too?
>
Good question. It's probably best to close the original entry as
"withdrawn" and I'll add a new entry. Sounds OK?
> 3. Patch is not calling interleave on PQ shmem, do we want to add that
> in as some next item like v1-0007? Question is whether OS interleaving
> makes sense there ? I believe it does there, please see my thread
> (NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
> being spawned by postmaster and may end up on different NUMA nodes
> randomly, so actually OS-interleaving that memory reduces jitter there
> (AKA bandwidth-over-latency). My thinking is that one cannot expect
> static/forced CPU-to-just-one-NUMA-node assignment for backend and
> it's PQ workers, because it is impossible have always available CPU
> power there in that NUMA node, so it might be useful to interleave
> that shared mem there too (as separate patch item?)
>
Excellent question. I haven't thought about this at all. I agree it
probably makes sense to interleave this memory, in some way. I don't
know what's the perfect scheme, though.
wild idea: Would it make sense to pin the workers to the same NUMA node
as the leader? And allocate all memory only from that node?
> 4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
> (also in v1-0005). My worry is should we may put some
> known-limitations docs (?) from start and mention that
> if the VM is greatly resized and NUMA numa nodes appear, they might
> not be used until restart?
>
Yes, this is one thing I need some feedback on. The patches mostly
assume there are no disabled nodes, that the set of allowed nodes does
not change, etc. I think for V1 that's a reasonable limitation.
But let's say we want to relax this a bit. How do we learn about the
change, after a node/CPU gets disabled? For some parts it's not that
difficult (e.g. we can "remap" buffers/descriptors) in the background.
But for other parts that's not practical. E.g. we can't rework how the
PGPROC gets split.
But while discussing this with Andres yesterday, he had an interesting
suggestion - to always use e.g. 8 or 16 partitions, then partition this
by NUMA node. So we'd have 16 partitions, and with 4 nodes the 0-3 would
go to node 0, 4-7 would go to node 1, etc. The advantage is that if a
node gets disabled, we can rebuild just this small "mapping" and not the
16 partitions. And the partitioning may be helpful even without NUMA.
Still have to figure out the details, but seems it might help.
> 5. In v1-0001, pg_numa_interleave_memory()
>
> + * XXX no return value, to make this fail on error, has to use
> + * numa_set_strict
>
> Yes, my patch has those numa_error() and numa_warn() handlers too in
> pg_numa. Feel free to use it for better UX.
>
> + * XXX Should we still touch the memory first, like
> with numa_move_pages,
> + * or is that not necessary?
>
> It's not necessary to touch after numa_tonode_memory() (wrapper around
> numa_interleave_memory()), if it is going to be used anyway it will be
> correctly placed to best of my knowledge.
>
> 6. diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
>
> Accidental indents (also fails to apply)
>
> 7. We miss the pg_numa_* shims, but for sure that's for later and also
> avoid those Linux specific #ifdef USE_LIBNUMA and so on?
>
Right, we need to add those. Or actually, we need to think about how
we'd do this for non-NUMA systems. I wonder if we even want to just
build everything the "old way" (without the partitions, etc.).
But per the earlier comment, the partitioning seems beneficial even on
non-NUMA systems, so maybe the shims are good enough OK.
> 8. v1-0005 2x + /* if (numa_procs_interleave) */
>
> Ha! it's a TRAP! I've uncommented it because I wanted to try it out
> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
>
> 2025-07-04 12:31:08.103 CEST [28754] LOG: starting PostgreSQL
> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
> [..]
> 2025-07-04 12:31:08.109 CEST [28754] LOG: io worker (PID 28755)
> was terminated by signal 11: Segmentation fault
> 2025-07-04 12:31:08.109 CEST [28754] LOG: terminating any other
> active server processes
> 2025-07-04 12:31:08.114 CEST [28754] LOG: shutting down because
> "restart_after_crash" is off
> 2025-07-04 12:31:08.116 CEST [28754] LOG: database system is shut down
>
> [New LWP 28755]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `postgres: io worker '.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> at ./nptl/sem_waitcommon.c:136
> 136 ./nptl/sem_waitcommon.c: No such file or directory.
> (gdb) where
> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> at ./nptl/sem_waitcommon.c:136
> #1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
> #2 0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
> ../src/backend/port/posix_sema.c:302
> #3 0x0000556191970553 in InitAuxiliaryProcess () at
> ../src/backend/storage/lmgr/proc.c:992
> #4 0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
> ../src/backend/postmaster/auxprocess.c:65
> #5 0x0000556191940676 in IoWorkerMain (startup_data=<optimized
> out>, startup_data_len=<optimized out>) at
> ../src/backend/storage/aio/method_worker.c:393
> #6 0x00005561918e8163 in postmaster_child_launch
> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
> startup_data=startup_data@entry=0x0,
> startup_data_len=startup_data_len@entry=0,
> client_sock=client_sock@entry=0x0) at
> ../src/backend/postmaster/launch_backend.c:290
> #7 0x00005561918ea09a in StartChildProcess
> (type=type@entry=B_IO_WORKER) at
> ../src/backend/postmaster/postmaster.c:3973
> #8 0x00005561918ea308 in maybe_adjust_io_workers () at
> ../src/backend/postmaster/postmaster.c:4404
> [..]
> (gdb) print *MyProc->sem
> Cannot access memory at address 0x0
>
Yeah, good catch. I'll look into that next week.
> 9. v1-0006: is this just a thought or serious candidate? I can imagine
> it can easily blow-up with some backends somehow requesting CPUs only
> from one NUMA node, while the second node being idle. Isn't it better
> just to leave CPU scheduling, well, to the CPU scheduler? The problem
> is that you have tools showing overall CPU usage, even mpstat(1) per
> CPU , but no tools for per-NUMA node CPU util%, so it would be hard
> for someone to realize that this is happening.
>
Mostly experimental, for benchmarking etc. I agree we may not want to
mess with the task scheduling too much.
Thanks for the feedback!
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-05 07:09 Cédric Villemain <[email protected]>
parent: Tomas Vondra <[email protected]>
3 siblings, 2 replies; 89+ messages in thread
From: Cédric Villemain @ 2025-07-05 07:09 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Tomas,
I haven't yet had time to fully read all the work and proposals around
NUMA and related features, but I hope to catch up over the summer.
However, I think it's important to share some thoughts before it's too
late, as you might find them relevant to the NUMA management code.
> 6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch
>
> This is an experimental patch, that simply pins the new process to the
> NUMA node obtained from the freelist.
>
> Driven by GUC "numa_procs_pin" (default: off).
In my work on more careful PostgreSQL resource management, I've come to
the conclusion that we should avoid pushing policy too deeply into the
PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
NUMA-specific management directly into core PostgreSQL in such a way.
We are working on a PROFILE and PROFILE MANAGER specification to provide
PostgreSQL with only the APIs and hooks needed so that extensions can
manage whatever they want externally.
The basic syntax (not meant to be discussed here, and even the names
might change) is roughly as follows, just to illustrate the intent:
CREATE PROFILE MANAGER manager_name [IF NOT EXISTS]
[ HANDLER handler_function | NO HANDLER ]
[ VALIDATOR validator_function | NO VALIDATOR ]
[ OPTIONS ( option 'value' [, ... ] ) ]
CREATE PROFILE profile_name
[IF NOT EXISTS]
USING profile_manager
SET key = value [, key = value]...
[USING profile_manager
SET key = value [, key = value]...]
[...];
CREATE PROFILE MAPPING
[IF NOT EXISTS]
FOR PROFILE profile_name
[MATCH [ ALL | ANY ] (
[ROLE role_name],
[BACKEND TYPE backend_type],
[DATABASE database_name],
[APPLICATION appname]
)];
## PROFILE RESOLUTION ORDER
1. ALTER ROLE IN DATABASE
2. ALTER ROLE
3. ALTER DATABASE
4. First matching PROFILE MAPPING (global or specific)
5. No profile (fallback)
As currently designed, this approach allows quite a lot of flexibility:
* pg_psi is used to ensure the spec is suitable for a cgroup profile
manager (moving PIDs as needed; NUMA and cgroups could work well
together, see e.g. this Linux kernel summary:
https://blogs.oracle.com/linux/post/numa-balancing )
* Someone else could implement support for Windows or BSD specifics.
* Others might use it to integrate PostgreSQL's own resources (e.g.,
"areas" of shared buffers) into policies.
Hope this perspective is helpful.
Best regards,
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-07 12:01 Tomas Vondra <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-07 12:01 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; PostgreSQL Hackers <[email protected]>
On 7/5/25 09:09, Cédric Villemain wrote:
> Hi Tomas,
>
>
> I haven't yet had time to fully read all the work and proposals around
> NUMA and related features, but I hope to catch up over the summer.
>
> However, I think it's important to share some thoughts before it's too
> late, as you might find them relevant to the NUMA management code.
>
>
>> 6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch
>>
>> This is an experimental patch, that simply pins the new process to the
>> NUMA node obtained from the freelist.
>>
>> Driven by GUC "numa_procs_pin" (default: off).
>
>
> In my work on more careful PostgreSQL resource management, I've come to
> the conclusion that we should avoid pushing policy too deeply into the
> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
> NUMA-specific management directly into core PostgreSQL in such a way.
>
>
> We are working on a PROFILE and PROFILE MANAGER specification to provide
> PostgreSQL with only the APIs and hooks needed so that extensions can
> manage whatever they want externally.
>
> The basic syntax (not meant to be discussed here, and even the names
> might change) is roughly as follows, just to illustrate the intent:
>
>
> CREATE PROFILE MANAGER manager_name [IF NOT EXISTS]
> [ HANDLER handler_function | NO HANDLER ]
> [ VALIDATOR validator_function | NO VALIDATOR ]
> [ OPTIONS ( option 'value' [, ... ] ) ]
>
> CREATE PROFILE profile_name
> [IF NOT EXISTS]
> USING profile_manager
> SET key = value [, key = value]...
> [USING profile_manager
> SET key = value [, key = value]...]
> [...];
>
> CREATE PROFILE MAPPING
> [IF NOT EXISTS]
> FOR PROFILE profile_name
> [MATCH [ ALL | ANY ] (
> [ROLE role_name],
> [BACKEND TYPE backend_type],
> [DATABASE database_name],
> [APPLICATION appname]
> )];
>
> ## PROFILE RESOLUTION ORDER
>
> 1. ALTER ROLE IN DATABASE
> 2. ALTER ROLE
> 3. ALTER DATABASE
> 4. First matching PROFILE MAPPING (global or specific)
> 5. No profile (fallback)
>
> As currently designed, this approach allows quite a lot of flexibility:
>
> * pg_psi is used to ensure the spec is suitable for a cgroup profile
> manager (moving PIDs as needed; NUMA and cgroups could work well
> together, see e.g. this Linux kernel summary: https://blogs.oracle.com/
> linux/post/numa-balancing )
>
> * Someone else could implement support for Windows or BSD specifics.
>
> * Others might use it to integrate PostgreSQL's own resources (e.g.,
> "areas" of shared buffers) into policies.
>
> Hope this perspective is helpful.
Can you explain how you want to manage this by an extension defined at
the SQL level, when most of this stuff has to be done when setting up
shared memory, which is waaaay before we have any access to catalogs?
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-07 12:31 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Jakub Wartak @ 2025-07-07 12:31 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi Tomas, some more thoughts after the weekend:
On Fri, Jul 4, 2025 at 8:12 PM Tomas Vondra <[email protected]> wrote:
>
> On 7/4/25 13:05, Jakub Wartak wrote:
> > On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <[email protected]> wrote:
> >
> > Hi!
> >
> >> 1) v1-0001-NUMA-interleaving-buffers.patch
> > [..]
> >> It's a bit more complicated, because the patch distributes both the
> >> blocks and descriptors, in the same way. So a buffer and it's descriptor
> >> always end on the same NUMA node. This is one of the reasons why we need
> >> to map larger chunks, because NUMA works on page granularity, and the
> >> descriptors are tiny - many fit on a memory page.
> >
> > Oh, now I get it! OK, let's stick to this one.
> >
> >> I don't think the splitting would actually make some things simpler, or
> >> maybe more flexible - in particular, it'd allow us to enable huge pages
> >> only for some regions (like shared buffers), and keep the small pages
> >> e.g. for PGPROC. So that'd be good.
> >
> > You have made assumption that this is good, but small pages (4KB) are
> > not hugetlb, and are *swappable* (Transparent HP are swappable too,
> > manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> > most frequent problem I see these days are OOMs, and it makes me
> > believe that making certain critical parts of shared memory being
> > swappable just to make pagesize granular is possibly throwing the baby
> > out with the bathwater. I'm thinking about bad situations like: some
> > wrong settings of vm.swapiness that people keep (or distros keep?) and
> > general inability of PG to restrain from allocating more memory in
> > some cases.
> >
>
> I haven't observed such issues myself, or maybe I didn't realize it's
> happening. Maybe it happens, but it'd be good to see some data showing
> that, or a reproducer of some sort. But let's say it's real.
>
> I don't think we should use huge pages merely to ensure something is not
> swapped out. The "not swappable" is more of a limitation of huge pages,
> not an advantage. You can't just choose to make them swappable.
>
> Wouldn't it be better to keep using 4KB pages, but lock the memory using
> mlock/mlockall?
In my book, not being swappable is a win (it's hard for me to imagine
when it could be beneficial to swap out parts of s_b).
I was trying to think about it and also got those:
Anyway mlock() probably sounds like it, but e.g. Rocky 8.10 by default
has max locked memory (ulimit -l) as low as 64kB due to systemd's
DefaultLimitMEMLOCK, but Debian/Ubuntu have those at higher values.
Wasn't expecting that - those are bizzare low values. I think we would
need something like (10000*900)/1024/1024 or more, but with each
PGPROC on a separate page that would be even way more?
Another thing with 4kB pages: there's this big assumption now made
that once we arrive in InitProcess() we won't ever change NUMA node,
so we stick to the PGPROC from where we started (based on getcpu(2)).
Let's assume CPU scheduler reassigned us to differnt node, but we have
now this 4kB patch ready for PGPROC in theory and this means we would
need to rely on the NUMA autobalancing doing it's job to migrate that
4kB page from node to node (to get better local accesses instead of
remote ones). The questions in my head are now like that:
- but we have asked intially asked those PGPROC pages to be localized
on certain node (they have policy), so they won't autobalance? We
would need to somewhere call getcpu() again notice the difference and
unlocalize (clear the NUMA/mbind() policy) for the PGPROC page?
- mlocked() as above says stick to physical RAM page (?) , so it won't move?
- after what time kernel's autobalancing would migrate that page since
switching the active CPU<->node? I mean do we execute enough reads on
this page?
BTW: to move this into pragmatic real, what's the most
one-liner/trivial way to exercise/stress PGPROC?
> >> The other thing I haven't thought about very much is determining on
> >> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> >> simply inherit/determine that at the start through libnuma, not through
> >> some custom PG configuration (which the patch [2] proposed to do).
> >
> > 0. I think that we could do better, some counter arguments to
> > no-configuration-at-all:
> >
> > a. as Robert & Bertrand already put it there after review: let's say I
> > want just to run on NUMA #2 node, so here I would need to override
> > systemd's script ExecStart= to include that numactl (not elegant?). I
> > could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
> > less friendly. Also it probably requires root to edit/reload systemd,
> > while having GUC for this like in my proposal makes it more smooth (I
> > think?)
> >
> > b. wouldn't it be better if that stayed as drop-in rather than always
> > on? What if there's a problem, how do you disable those internal
> > optimizations if they do harm in some cases? (or let's say I want to
> > play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
> > numa_buffers_interleave would be nice?
> >
> > c. What if I want my standby (walreceiver+startup/recovery) to run
> > with NUMA affinity to get better performance (I'm not going to hack
> > around systemd script every time, but I could imagine changing
> > numa=X,Y,Z after restart/before promotion)
> >
> > d. Now if I would be forced for some reason to do that numactl(1)
> > voodoo, and use the those above mentioned overrides and PG wouldn't be
> > having GUC (let's say I would use `numactl
> > --weighted-interleave=0,1`), then:
> >
>
> I'm not against doing something like this, but I don't plan to do that
> in V1. I don't have a clear idea what configurability is actually
> needed, so it's likely I'd do the interface wrong.
>
> >> 2) v1-0002-NUMA-localalloc.patch
> >> This simply sets "localalloc" when initializing a backend, so that all
> >> memory allocated later is local, not interleaved. Initially this was
> >> necessary because the patch set the allocation policy to interleaving
> >> before initializing shared memory, and we didn't want to interleave the
> >> private memory. But that's no longer the case - the explicit mapping to
> >> nodes does not have this issue. I'm keeping the patch for convenience,
> >> it allows experimenting with numactl etc.
> >
> > .. .is not accurate anymore and we would require to have that in
> > (still with GUC) ?
> > Thoughts? I can add that mine part into Your's patches if you want.
> >
>
> I'm sorry, I don't understand what's the question :-(
That patch reference above, it was a chain of thought from step "d".
What I had in mind was that you cannot remove the patch
`v1-0002-NUMA-localalloc.patch` from the scope if forcing people to
use numactl by not having enough configurability on the PG side. That
is: if someone will have to use systemd+numactl
--interleave/--weighted-interleave then, he will also need to have a
way to use numa_localalloc=on (to override the new/user's policy
default, otherwise local mem allocations are also going to be
interleaved, and we are back to square one). Which brings me to a
point why instead of this toggle, should include the configuration
properly inside from start (it's not that hard apparently).
> > Way too quick review and some very fast benchmark probes, I've
> > concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
> > would be too new topic for me), but let's start:
> >
> > 1. normal pgbench -S (still with just s_b@4GB), done many tries,
> > consistent benefit for the patch with like +8..10% boost on generic
> > run:
> >
[.. removed numbers]
>
> But this actually brings an interesting question. What exactly should we
> expect / demand from these patches? In my mind it'd primarily about
> predictability and stability of results.
>
> For example, the results should not depend on how was the database
> warmed up - was it done by a single backend or many backends? Was it
> restarted, or what? I could probably warmup the system very carefully to
> ensure it's balanced. The patches mean I don't need to be that careful.
Well, pretty much the same here. I was after minimizing "stddev" (to
have better predictability of results, especially across restarts) and
increasing available bandwidth [which is pretty much related]. Without
our NUMA work, PG can just put that s_b on any random node or spill
randomly from to another (depending on size of allocation request).
> > So should I close https://commitfest.postgresql.org/patch/5703/
> > and you'll open a new one or should I just edit the #5703 and alter it
> > and add this thread too?
> >
>
> Good question. It's probably best to close the original entry as
> "withdrawn" and I'll add a new entry. Sounds OK?
Sure thing, marked it as `Returned with feedback`, this approach seems
to be much more advanced.
> > 3. Patch is not calling interleave on PQ shmem, do we want to add that
> > in as some next item like v1-0007? Question is whether OS interleaving
> > makes sense there ? I believe it does there, please see my thread
> > (NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
> > being spawned by postmaster and may end up on different NUMA nodes
> > randomly, so actually OS-interleaving that memory reduces jitter there
> > (AKA bandwidth-over-latency). My thinking is that one cannot expect
> > static/forced CPU-to-just-one-NUMA-node assignment for backend and
> > it's PQ workers, because it is impossible have always available CPU
> > power there in that NUMA node, so it might be useful to interleave
> > that shared mem there too (as separate patch item?)
> >
>
> Excellent question. I haven't thought about this at all. I agree it
> probably makes sense to interleave this memory, in some way. I don't
> know what's the perfect scheme, though.
>
> wild idea: Would it make sense to pin the workers to the same NUMA node
> as the leader? And allocate all memory only from that node?
I'm trying to convey exactly the opposite message or at least that it
might depend on configuration. Please see
https://www.postgresql.org/message-id/CAKZiRmxYMPbQ4WiyJWh%3DVuw_Ny%2BhLGH9_9FaacKRJvzZ-smm%2Bw%40ma...
(btw it should read there that I don't indent spend a lot of thime on
PQ), but anyway: I think we should NOT pin the PQ workers the same
NODE as you do not know if there's CPU left there (same story as with
v1-0006 here).
I'm just proposing quick OS-based interleaving of PQ shm if using all
nodes, literally:
@@ -334,6 +336,13 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
request_size,
}
*mapped_address = address;
*mapped_size = request_size;
+
+ /* We interleave memory only at creation time. */
+ if (op == DSM_OP_CREATE && numa->setting > NUMA_OFF) {
+ elog(DEBUG1, "interleaving shm mem @ %p size=%zu",
*mapped_address, *mapped_size);
+ pg_numa_interleave_memptr(*mapped_address, *mapped_size, numa->nodes);
+ }
+
Because then if memory is interleaved you have probably less variance
for memory access. But also from that previous thread:
"So if anything:
- latency-wise: it would be best to place leader+all PQ workers close
to s_b, provided s_b fits NUMA shared/huge page memory there and you
won't need more CPU than there's on that NUMA node... (assuming e.g.
hosting 4 DBs on 4-sockets each on it's own, it would be best to pin
everything including shm, but PQ workers too)
- capacity/TPS-wise or s_b > NUMA: just interleave to maximize
bandwidth and get uniform CPU performance out of this"
So wild idea was: maybe PQ shm interleaving should on NUMA
configuration (if intereavling to all nodes, then interleave normally,
but if configuration sets to just 1 NUMA node, it automatically binds
there -- there was '@' support for that in my patch).
> > 4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
> > (also in v1-0005). My worry is should we may put some
> > known-limitations docs (?) from start and mention that
> > if the VM is greatly resized and NUMA numa nodes appear, they might
> > not be used until restart?
> >
>
> Yes, this is one thing I need some feedback on. The patches mostly
> assume there are no disabled nodes, that the set of allowed nodes does
> not change, etc. I think for V1 that's a reasonable limitation.
Sure!
> But let's say we want to relax this a bit. How do we learn about the
> change, after a node/CPU gets disabled? For some parts it's not that
> difficult (e.g. we can "remap" buffers/descriptors) in the background.
> But for other parts that's not practical. E.g. we can't rework how the
> PGPROC gets split.
>
> But while discussing this with Andres yesterday, he had an interesting
> suggestion - to always use e.g. 8 or 16 partitions, then partition this
> by NUMA node. So we'd have 16 partitions, and with 4 nodes the 0-3 would
> go to node 0, 4-7 would go to node 1, etc. The advantage is that if a
> node gets disabled, we can rebuild just this small "mapping" and not the
> 16 partitions. And the partitioning may be helpful even without NUMA.
>
> Still have to figure out the details, but seems it might help.
Right, no idea how the shared_memory remapping patch will work
(how/when the s_b change will be executed), but we could somehow mark
that number of NUMA zones could be rechecked during SIGHUP (?) and
then just simple compare check if old_numa_num_configured_nodes ==
new_numa_num_configured_nodes is true.
Anyway, I think it's way too advanced for now, don't you think? (like
CPU ballooning [s_b itself] is rare, and NUMA ballooning seems to be
super-wild-rare).
As for the rest, forgot to include this too: getcpu() - this really
needs a portable pg_getcpu() wrapper.
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-07 14:51 Cédric Villemain <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Cédric Villemain @ 2025-07-07 14:51 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>> "areas" of shared buffers) into policies.
>>
>> Hope this perspective is helpful.
>
> Can you explain how you want to manage this by an extension defined at
> the SQL level, when most of this stuff has to be done when setting up
> shared memory, which is waaaay before we have any access to catalogs?
I should have said module instead, I didn't follow carefully but at some
point there were discussion about shared buffers resized "on-line".
Anyway, it was just to give some few examples, maybe this one is to be
considered later (I'm focused on cgroup/psi, and precisely reassigning
PIDs as needed).
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-07 22:35 Tomas Vondra <[email protected]>
parent: Cédric Villemain <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-07 22:35 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; PostgreSQL Hackers <[email protected]>
On 7/7/25 16:51, Cédric Villemain wrote:
>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>> "areas" of shared buffers) into policies.
>>>
>>> Hope this perspective is helpful.
>>
>> Can you explain how you want to manage this by an extension defined at
>> the SQL level, when most of this stuff has to be done when setting up
>> shared memory, which is waaaay before we have any access to catalogs?
>
> I should have said module instead, I didn't follow carefully but at some
> point there were discussion about shared buffers resized "on-line".
> Anyway, it was just to give some few examples, maybe this one is to be
> considered later (I'm focused on cgroup/psi, and precisely reassigning
> PIDs as needed).
>
I don't know. I have a hard time imagining what exactly would the
policies / profiles do exactly to respond to changes in the system
utilization. And why should that interfere with this patch ...
The main thing patch series aims to implement is partitioning different
pieces of shared memory (buffers, freelists, ...) to better work for
NUMA. I don't think there's that many ways to do this, and I doubt it
makes sense to make this easily customizable from external modules of
any kind. I can imagine providing some API allowing to isolate the
instance on selected NUMA nodes, but that's about it.
Yes, there's some relation to the online resizing of shared buffers, in
which case we need to "refresh" some of the information. But AFAICS it's
not very extensive (on top of what already needs to happen after the
resize), and it'd happen within the boundaries of the partitioning
scheme. There's not that much flexibility.
The last bit (pinning backends to a NUMA node) is experimental, and
mostly intended for easier evaluation of the earlier parts (e.g. to
limit the noise when processes get moved to a CPU from a different NUMA
node, and so on).
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-07 23:25 Andres Freund <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-07 23:25 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
> In my work on more careful PostgreSQL resource management, I've come to the
> conclusion that we should avoid pushing policy too deeply into the
> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
> NUMA-specific management directly into core PostgreSQL in such a way.
I think it's actually the opposite - whenever we pushed stuff like this
outside of core it has hurt postgres substantially. Not having replication in
core was a huge mistake. Not having HA management in core is probably the
biggest current adoption hurdle for postgres.
To deal better with NUMA we need to improve memory placement and various
algorithms, in an interrelated way - that's pretty much impossible to do
outside of core.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 01:47 Cédric Villemain <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Cédric Villemain @ 2025-07-08 01:47 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
> On 7/7/25 16:51, Cédric Villemain wrote:
>>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>>> "areas" of shared buffers) into policies.
>>>>
>>>> Hope this perspective is helpful.
>>>
>>> Can you explain how you want to manage this by an extension defined at
>>> the SQL level, when most of this stuff has to be done when setting up
>>> shared memory, which is waaaay before we have any access to catalogs?
>>
>> I should have said module instead, I didn't follow carefully but at some
>> point there were discussion about shared buffers resized "on-line".
>> Anyway, it was just to give some few examples, maybe this one is to be
>> considered later (I'm focused on cgroup/psi, and precisely reassigning
>> PIDs as needed).
>>
>
> I don't know. I have a hard time imagining what exactly would the
> policies / profiles do exactly to respond to changes in the system
> utilization. And why should that interfere with this patch ...
>
> The main thing patch series aims to implement is partitioning different
> pieces of shared memory (buffers, freelists, ...) to better work for
> NUMA. I don't think there's that many ways to do this, and I doubt it
> makes sense to make this easily customizable from external modules of
> any kind. I can imagine providing some API allowing to isolate the
> instance on selected NUMA nodes, but that's about it.
>
> Yes, there's some relation to the online resizing of shared buffers, in
> which case we need to "refresh" some of the information. But AFAICS it's
> not very extensive (on top of what already needs to happen after the
> resize), and it'd happen within the boundaries of the partitioning
> scheme. There's not that much flexibility.
>
> The last bit (pinning backends to a NUMA node) is experimental, and
> mostly intended for easier evaluation of the earlier parts (e.g. to
> limit the noise when processes get moved to a CPU from a different NUMA
> node, and so on).
The backend pinning can be done by replacing your patch on proc.c to
call an external profile manager doing exactly the same thing maybe ?
Similar to:
pmroutine = GetPmRoutineForInitProcess();
if (pmroutine != NULL &&
pmroutine->init_process != NULL)
pmroutine->init_process(MyProc);
...
pmroutine = GetPmRoutineForInitAuxilliary();
if (pmroutine != NULL &&
pmroutine->init_auxilliary != NULL)
pmroutine->init_auxilliary(MyProc);
Added on some rare places should cover most if not all the requirement
around process placement (process_shared_preload_libraries() is called
earlier in the process creation I believe).
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 01:55 Cédric Villemain <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Cédric Villemain @ 2025-07-08 01:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Andres,
> Hi,
>
> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>> In my work on more careful PostgreSQL resource management, I've come to the
>> conclusion that we should avoid pushing policy too deeply into the
>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>> NUMA-specific management directly into core PostgreSQL in such a way.
>
> I think it's actually the opposite - whenever we pushed stuff like this
> outside of core it has hurt postgres substantially. Not having replication in
> core was a huge mistake. Not having HA management in core is probably the
> biggest current adoption hurdle for postgres.
>
> To deal better with NUMA we need to improve memory placement and various
> algorithms, in an interrelated way - that's pretty much impossible to do
> outside of core.
Except the backend pinning which is easy to achieve, thus my comment on
the related patch.
I'm not claiming NUMA memory and all should be managed outside of core
(though I didn't read other patches yet).
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 02:14 Cédric Villemain <[email protected]>
parent: Cédric Villemain <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Cédric Villemain @ 2025-07-08 02:14 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
>> On 7/7/25 16:51, Cédric Villemain wrote:
>>>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>>>> "areas" of shared buffers) into policies.
>>>>>
>>>>> Hope this perspective is helpful.
>>>>
>>>> Can you explain how you want to manage this by an extension defined at
>>>> the SQL level, when most of this stuff has to be done when setting up
>>>> shared memory, which is waaaay before we have any access to catalogs?
>>>
>>> I should have said module instead, I didn't follow carefully but at some
>>> point there were discussion about shared buffers resized "on-line".
>>> Anyway, it was just to give some few examples, maybe this one is to be
>>> considered later (I'm focused on cgroup/psi, and precisely reassigning
>>> PIDs as needed).
>>>
>>
>> I don't know. I have a hard time imagining what exactly would the
>> policies / profiles do exactly to respond to changes in the system
>> utilization. And why should that interfere with this patch ...
>>
>> The main thing patch series aims to implement is partitioning different
>> pieces of shared memory (buffers, freelists, ...) to better work for
>> NUMA. I don't think there's that many ways to do this, and I doubt it
>> makes sense to make this easily customizable from external modules of
>> any kind. I can imagine providing some API allowing to isolate the
>> instance on selected NUMA nodes, but that's about it.
>>
>> Yes, there's some relation to the online resizing of shared buffers, in
>> which case we need to "refresh" some of the information. But AFAICS it's
>> not very extensive (on top of what already needs to happen after the
>> resize), and it'd happen within the boundaries of the partitioning
>> scheme. There's not that much flexibility.
>>
>> The last bit (pinning backends to a NUMA node) is experimental, and
>> mostly intended for easier evaluation of the earlier parts (e.g. to
>> limit the noise when processes get moved to a CPU from a different NUMA
>> node, and so on).
>
> The backend pinning can be done by replacing your patch on proc.c to
> call an external profile manager doing exactly the same thing maybe ?
>
> Similar to:
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
> pmroutine->init_process != NULL)
> pmroutine->init_process(MyProc);
>
> ...
>
> pmroutine = GetPmRoutineForInitAuxilliary();
> if (pmroutine != NULL &&
> pmroutine->init_auxilliary != NULL)
> pmroutine->init_auxilliary(MyProc);
>
> Added on some rare places should cover most if not all the requirement
> around process placement (process_shared_preload_libraries() is called
> earlier in the process creation I believe).
>
After a first read I think this works for patches 002 and 005. For this
last one, InitProcGlobal() may setup things as you do but then expose
the choice a bit later, basically in places where you added the if
condition on the GUC: numa_procs_interleave).
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-08 03:04 Andres Freund <[email protected]>
parent: Jakub Wartak <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-08 03:04 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <[email protected]> wrote:
> > I don't think the splitting would actually make some things simpler, or
> > maybe more flexible - in particular, it'd allow us to enable huge pages
> > only for some regions (like shared buffers), and keep the small pages
> > e.g. for PGPROC. So that'd be good.
>
> You have made assumption that this is good, but small pages (4KB) are
> not hugetlb, and are *swappable* (Transparent HP are swappable too,
> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> most frequent problem I see these days are OOMs, and it makes me
> believe that making certain critical parts of shared memory being
> swappable just to make pagesize granular is possibly throwing the baby
> out with the bathwater. I'm thinking about bad situations like: some
> wrong settings of vm.swapiness that people keep (or distros keep?) and
> general inability of PG to restrain from allocating more memory in
> some cases.
The reason it would be advantageous to put something like the procarray onto
smaller pages is that otherwise the entire procarray (unless particularly
large) ends up on a single NUMA node, increasing the latency for backends on
every other numa node and increasing memory traffic on that node.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-08 12:27 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-08 12:27 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/8/25 05:04, Andres Freund wrote:
> Hi,
>
> On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
>> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <[email protected]> wrote:
>>> I don't think the splitting would actually make some things simpler, or
>>> maybe more flexible - in particular, it'd allow us to enable huge pages
>>> only for some regions (like shared buffers), and keep the small pages
>>> e.g. for PGPROC. So that'd be good.
>>
>> You have made assumption that this is good, but small pages (4KB) are
>> not hugetlb, and are *swappable* (Transparent HP are swappable too,
>> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
>> most frequent problem I see these days are OOMs, and it makes me
>> believe that making certain critical parts of shared memory being
>> swappable just to make pagesize granular is possibly throwing the baby
>> out with the bathwater. I'm thinking about bad situations like: some
>> wrong settings of vm.swapiness that people keep (or distros keep?) and
>> general inability of PG to restrain from allocating more memory in
>> some cases.
>
> The reason it would be advantageous to put something like the procarray onto
> smaller pages is that otherwise the entire procarray (unless particularly
> large) ends up on a single NUMA node, increasing the latency for backends on
> every other numa node and increasing memory traffic on that node.
>
That's why the patch series splits the procarray into multiple pieces,
so that it can be properly distributed on multiple NUMA nodes even with
huge pages. It requires adjusting a couple places accessing the entries,
but it surprised me how limited the impact was.
If we could selectively use 4KB pages for parts of the shared memory,
maybe this wouldn't be necessary. But it's not too annoying.
The thing I'm not sure about is how much this actually helps with the
traffic between node. Sure, if we pick a PGPROC from the same node, and
the task does not get moved, it'll be local traffic. But if the task
moves, there'll be traffic. I don't have any estimates how often this
happens, e.g. for older tasks.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 12:34 Tomas Vondra <[email protected]>
parent: Cédric Villemain <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-08 12:34 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/8/25 03:55, Cédric Villemain wrote:
> Hi Andres,
>
>> Hi,
>>
>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>> In my work on more careful PostgreSQL resource management, I've come
>>> to the
>>> conclusion that we should avoid pushing policy too deeply into the
>>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>
>> I think it's actually the opposite - whenever we pushed stuff like this
>> outside of core it has hurt postgres substantially. Not having
>> replication in
>> core was a huge mistake. Not having HA management in core is probably the
>> biggest current adoption hurdle for postgres.
>>
>> To deal better with NUMA we need to improve memory placement and various
>> algorithms, in an interrelated way - that's pretty much impossible to do
>> outside of core.
>
> Except the backend pinning which is easy to achieve, thus my comment on
> the related patch.
> I'm not claiming NUMA memory and all should be managed outside of core
> (though I didn't read other patches yet).
>
But an "optimal backend placement" seems to very much depend on where we
placed the various pieces of shared memory. Which the external module
will have trouble following, I suspect.
I still don't have any idea what exactly would the external module do,
how would it decide where to place the backend. Can you describe some
use case with an example?
Assuming we want to actually pin tasks from within Postgres, what I
think might work is allowing modules to "advise" on where to place the
task. But the decision would still be done by core.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-08 12:56 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-08 12:56 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> On 7/8/25 05:04, Andres Freund wrote:
> > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > The reason it would be advantageous to put something like the procarray onto
> > smaller pages is that otherwise the entire procarray (unless particularly
> > large) ends up on a single NUMA node, increasing the latency for backends on
> > every other numa node and increasing memory traffic on that node.
> >
>
> That's why the patch series splits the procarray into multiple pieces,
> so that it can be properly distributed on multiple NUMA nodes even with
> huge pages. It requires adjusting a couple places accessing the entries,
> but it surprised me how limited the impact was.
Sure, you can do that, but it does mean that iterations over the procarray now
have an added level of indirection...
> The thing I'm not sure about is how much this actually helps with the
> traffic between node. Sure, if we pick a PGPROC from the same node, and
> the task does not get moved, it'll be local traffic. But if the task
> moves, there'll be traffic. I don't have any estimates how often this
> happens, e.g. for older tasks.
I think the most important bit is to not put everything onto one numa node,
otherwise the chance of increased latency for *everyone* due to the increased
memory contention is more likely to hurt.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 16:06 Cédric Villemain <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 2 replies; 89+ messages in thread
From: Cédric Villemain @ 2025-07-08 16:06 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
> On 7/8/25 03:55, Cédric Villemain wrote:
>> Hi Andres,
>>
>>> Hi,
>>>
>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>> In my work on more careful PostgreSQL resource management, I've come
>>>> to the
>>>> conclusion that we should avoid pushing policy too deeply into the
>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>
>>> I think it's actually the opposite - whenever we pushed stuff like this
>>> outside of core it has hurt postgres substantially. Not having
>>> replication in
>>> core was a huge mistake. Not having HA management in core is probably the
>>> biggest current adoption hurdle for postgres.
>>>
>>> To deal better with NUMA we need to improve memory placement and various
>>> algorithms, in an interrelated way - that's pretty much impossible to do
>>> outside of core.
>>
>> Except the backend pinning which is easy to achieve, thus my comment on
>> the related patch.
>> I'm not claiming NUMA memory and all should be managed outside of core
>> (though I didn't read other patches yet).
>>
>
> But an "optimal backend placement" seems to very much depend on where we
> placed the various pieces of shared memory. Which the external module
> will have trouble following, I suspect.
>
> I still don't have any idea what exactly would the external module do,
> how would it decide where to place the backend. Can you describe some
> use case with an example?
>
> Assuming we want to actually pin tasks from within Postgres, what I
> think might work is allowing modules to "advise" on where to place the
> task. But the decision would still be done by core.
Possibly exactly what you're doing in proc.c when managing allocation of
process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
candidates), I didn't get that they require information not available to
any process executing code from a module.
Parts of your code where you assign/define policy could be in one or
more relevant routines of a "numa profile manager", like in an
initProcessRoutine(), and registered in pmroutine struct:
pmroutine = GetPmRoutineForInitProcess();
if (pmroutine != NULL &&
pmroutine->init_process != NULL)
pmroutine->init_process(MyProc);
This way it's easier to manage alternative policies, and also to be able
to adjust when hardware and linux kernel changes.
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-08 21:26 Tomas Vondra <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-08 21:26 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/8/25 18:06, Cédric Villemain wrote:
>
>
>
>
>
>
>> On 7/8/25 03:55, Cédric Villemain wrote:
>>> Hi Andres,
>>>
>>>> Hi,
>>>>
>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>> to the
>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>> integrating
>>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>>
>>>> I think it's actually the opposite - whenever we pushed stuff like this
>>>> outside of core it has hurt postgres substantially. Not having
>>>> replication in
>>>> core was a huge mistake. Not having HA management in core is
>>>> probably the
>>>> biggest current adoption hurdle for postgres.
>>>>
>>>> To deal better with NUMA we need to improve memory placement and
>>>> various
>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>> to do
>>>> outside of core.
>>>
>>> Except the backend pinning which is easy to achieve, thus my comment on
>>> the related patch.
>>> I'm not claiming NUMA memory and all should be managed outside of core
>>> (though I didn't read other patches yet).
>>>
>>
>> But an "optimal backend placement" seems to very much depend on where we
>> placed the various pieces of shared memory. Which the external module
>> will have trouble following, I suspect.
>>
>> I still don't have any idea what exactly would the external module do,
>> how would it decide where to place the backend. Can you describe some
>> use case with an example?
>>
>> Assuming we want to actually pin tasks from within Postgres, what I
>> think might work is allowing modules to "advise" on where to place the
>> task. But the decision would still be done by core.
>
> Possibly exactly what you're doing in proc.c when managing allocation of
> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
> candidates), I didn't get that they require information not available to
> any process executing code from a module.
>
Well, it needs to understand how some other stuff (especially PGPROC
entries) is distributed between nodes. I'm not sure how much of this
internal information we want to expose outside core ...
> Parts of your code where you assign/define policy could be in one or
> more relevant routines of a "numa profile manager", like in an
> initProcessRoutine(), and registered in pmroutine struct:
>
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
> pmroutine->init_process != NULL)
> pmroutine->init_process(MyProc);
>
> This way it's easier to manage alternative policies, and also to be able
> to adjust when hardware and linux kernel changes.
>
I'm not against making this extensible, in some way. But I still
struggle to imagine a reasonable alternative policy, where the external
module gets the same information and ends up with a different decision.
So what would the alternate policy look like? What use case would the
module be supporting?
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-09 06:40 Cédric Villemain <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 2 replies; 89+ messages in thread
From: Cédric Villemain @ 2025-07-09 06:40 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
> On 7/8/25 18:06, Cédric Villemain wrote:
>>
>>
>>
>>
>>
>>
>>> On 7/8/25 03:55, Cédric Villemain wrote:
>>>> Hi Andres,
>>>>
>>>>> Hi,
>>>>>
>>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>>> to the
>>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>>> integrating
>>>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>>>
>>>>> I think it's actually the opposite - whenever we pushed stuff like this
>>>>> outside of core it has hurt postgres substantially. Not having
>>>>> replication in
>>>>> core was a huge mistake. Not having HA management in core is
>>>>> probably the
>>>>> biggest current adoption hurdle for postgres.
>>>>>
>>>>> To deal better with NUMA we need to improve memory placement and
>>>>> various
>>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>>> to do
>>>>> outside of core.
>>>>
>>>> Except the backend pinning which is easy to achieve, thus my comment on
>>>> the related patch.
>>>> I'm not claiming NUMA memory and all should be managed outside of core
>>>> (though I didn't read other patches yet).
>>>>
>>>
>>> But an "optimal backend placement" seems to very much depend on where we
>>> placed the various pieces of shared memory. Which the external module
>>> will have trouble following, I suspect.
>>>
>>> I still don't have any idea what exactly would the external module do,
>>> how would it decide where to place the backend. Can you describe some
>>> use case with an example?
>>>
>>> Assuming we want to actually pin tasks from within Postgres, what I
>>> think might work is allowing modules to "advise" on where to place the
>>> task. But the decision would still be done by core.
>>
>> Possibly exactly what you're doing in proc.c when managing allocation of
>> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
>> candidates), I didn't get that they require information not available to
>> any process executing code from a module.
>>
>
> Well, it needs to understand how some other stuff (especially PGPROC
> entries) is distributed between nodes. I'm not sure how much of this
> internal information we want to expose outside core ...
>
>> Parts of your code where you assign/define policy could be in one or
>> more relevant routines of a "numa profile manager", like in an
>> initProcessRoutine(), and registered in pmroutine struct:
>>
>> pmroutine = GetPmRoutineForInitProcess();
>> if (pmroutine != NULL &&
>> pmroutine->init_process != NULL)
>> pmroutine->init_process(MyProc);
>>
>> This way it's easier to manage alternative policies, and also to be able
>> to adjust when hardware and linux kernel changes.
>>
>
> I'm not against making this extensible, in some way. But I still
> struggle to imagine a reasonable alternative policy, where the external
> module gets the same information and ends up with a different decision.
>
> So what would the alternate policy look like? What use case would the
> module be supporting?
That's the whole point: there are very distinct usages of PostgreSQL in
the field. And maybe not all of them will require the policy defined by
PostgreSQL core.
May I ask the reverse: what prevent external modules from taking those
decisions ? There are already a lot of area where external code can take
over PostgreSQL processing, like Neon is doing.
There are some very early processing for memory setup that I can see as
a current blocker, and here I'd refer a more compliant NUMA api as
proposed by Jakub so it's possible to arrange based on workload,
hardware configuration or other matters. Reworking to get distinct
segment and all as you do is great, and combo of both approach probably
of great interest. There is also this weighted interleave discussed and
probably much more to come in this area in Linux.
I think some points raised already about possible distinct policies, I
am precisely claiming that it is hard to come with one good policy with
limited setup options, thus requirement to keep that flexible enough
(hooks, api, 100 GUc ?).
There is an EPYC story here also, given the NUMA setup can vary
depending on BIOS setup, associated NUMA policy must probably take that
into account (L3 can be either real cache or 4 extra "local" NUMA nodes
- with highly distinct access cost from a RAM module).
Does that change how PostgreSQL will place memory and process? Is it
important or of interest ?
--
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-09 08:09 Bertrand Drouvot <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Bertrand Drouvot @ 2025-07-09 08:09 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Andres Freund <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Wed, Jul 09, 2025 at 06:40:00AM +0000, Cédric Villemain wrote:
> > On 7/8/25 18:06, Cédric Villemain wrote:
> > I'm not against making this extensible, in some way. But I still
> > struggle to imagine a reasonable alternative policy, where the external
> > module gets the same information and ends up with a different decision.
> >
> > So what would the alternate policy look like? What use case would the
> > module be supporting?
>
>
> That's the whole point: there are very distinct usages of PostgreSQL in the
> field. And maybe not all of them will require the policy defined by
> PostgreSQL core.
>
> May I ask the reverse: what prevent external modules from taking those
> decisions ? There are already a lot of area where external code can take
> over PostgreSQL processing, like Neon is doing.
>
> There are some very early processing for memory setup that I can see as a
> current blocker, and here I'd refer a more compliant NUMA api as proposed by
> Jakub so it's possible to arrange based on workload, hardware configuration
> or other matters. Reworking to get distinct segment and all as you do is
> great, and combo of both approach probably of great interest.
I think that Tomas's approach helps to have more "predictable" performance
expectations, I mean more consistent over time, fewer "surprises".
While your approach (and Jakub's one)) could help to get performance gains
depending on a "known" context (so less generic).
So, probably having both could make sense but I think that they serve different
purposes.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 10:04 Jakub Wartak <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-07-09 10:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jul 8, 2025 at 2:56 PM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> > On 7/8/25 05:04, Andres Freund wrote:
> > > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > > The reason it would be advantageous to put something like the procarray onto
> > > smaller pages is that otherwise the entire procarray (unless particularly
> > > large) ends up on a single NUMA node, increasing the latency for backends on
> > > every other numa node and increasing memory traffic on that node.
> > >
Sure thing, I fully understand the motivation and underlying reason
(without claiming that I understand the exact memory access patterns
that involve procarray/PGPROC/etc and hotspots involved from PG side).
Any single-liner pgbench help for how to really easily stress the
PGPROC or procarray?
> > That's why the patch series splits the procarray into multiple pieces,
> > so that it can be properly distributed on multiple NUMA nodes even with
> > huge pages. It requires adjusting a couple places accessing the entries,
> > but it surprised me how limited the impact was.
Yes, and we are discussing if it is worth getting into smaller pages
for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
what more even more waste 1GB hugetlb if we dont request 2MB for some
small structs: btw, we have ability to select MAP_HUGE_2MB vs
MAP_HUGE_1GB). I'm thinking about two problems:
- 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning
- using libnuma often leads to MPOL_BIND which disarms NUMA
autobalancing, BUT apparently there are set_mempolicy(2)/mbind(2) and
since 5.12+ kernel they can take additional flag
MPOL_F_NUMA_BALANCING(!), so this looks like it has potential to move
memory anyway (if way too many tasks are relocated, so would be
memory?). It is available only in recent libnuma as
numa_set_membind_balancing(3), but sadly there's no way via libnuma to
do mbind(MPOL_F_NUMA_BALANCING) for a specific addr only? I mean it
would have be something like MPOL_F_NUMA_BALANCING | MPOL_PREFERRED?
(select one node from many for each node while still allowing
balancing?), but in [1][2] (2024) it is stated that "It's not
legitimate (yet) to use MPOL_PREFERRED + MPOL_F_NUMA_BALANCING.", but
maybe stuff has been improved since then.
Something like:
PGPROC/procarray 2MB page for node#1 - mbind(addr1,
MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [0,1]);
PGPROC/procarray 2MB page for node#2 - mbind(addr2,
MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [1,0]);
> Sure, you can do that, but it does mean that iterations over the procarray now
> have an added level of indirection...
So the most efficient would be the old-way (no indirections) vs
NUMA-way? Can this be done without #ifdefs at all?
> > The thing I'm not sure about is how much this actually helps with the
> > traffic between node. Sure, if we pick a PGPROC from the same node, and
> > the task does not get moved, it'll be local traffic. But if the task
> > moves, there'll be traffic.
With MPOL_F_NUMA_BALANCING, that should "auto-tune" in the worst case?
> > I don't have any estimates how often this happens, e.g. for older tasks.
We could measure, kernel 6.16+ has per PID numa_task_migrated in
/proc/{PID}/sched , but I assume we would have to throw backends >>
VCPUs at it, to simulate reality and do some "waves" between different
activity periods of certain pools (I can imagine worst case scenario:
a) pgbench "a" open $VCPU connections, all idle, with scripto to sleep
for a while
b) pgbench "b" open some $VCPU new connections to some other DB, all
active from start (tpcbb or readonly)
c) manually ping CPUs using taskset for each PID all from "b" to
specific NUMA node #2 -- just to simulate unfortunate app working on
every 2nd conn
d) pgbench "a" starts working and hits CPU imbalance -- e.g. NUMA node
#1 is idle, #2 is full, CPU scheduler starts puting "a" backends on
CPUs from #1 , and we should notice PIDs being migrated)
> I think the most important bit is to not put everything onto one numa node,
> otherwise the chance of increased latency for *everyone* due to the increased
> memory contention is more likely to hurt.
-J.
p.s. I hope i did write in an understandable way, because I had many
interruptions, so if anything is unclear please let me know.
[1] - https://lkml.org/lkml/2024/7/3/352
[2] - https://lkml.rescloud.iu.edu/2402.2/03227.html
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 16:35 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-09 16:35 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
On 2025-07-02 14:36:31 +0200, Tomas Vondra wrote:
> On 7/2/25 13:37, Ashutosh Bapat wrote:
> > On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <[email protected]> wrote:
> >>
> >>
> >> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
> >>
> >> Minor optimization. Andres noticed we're tracking the tail of buffer
> >> freelist, without using it. So the patch removes that.
> >>
> >
> > The patches for resizing buffers use the lastFreeBuffer to add new
> > buffers to the end of free list when expanding it. But we could as
> > well add it at the beginning of the free list.
Yea, I don't see any point in adding buffers to the tail instead of to the
front. We probably want more recently used buffers at the front, since they
(and the associated BufferDesc) are more likely to be in a CPU cache.
> > This patch seems almost independent of the rest of the patches. Do you
> > need it in the rest of the patches? I understand that those patches
> > don't need to worry about maintaining lastFreeBuffer after this patch.
> > Is there any other effect?
> >
> > If we are going to do this, let's do it earlier so that buffer
> > resizing patches can be adjusted.
> >
>
> My patches don't particularly rely on this bit, it would work even with
> lastFreeBuffer. I believe Andres simply noticed the current code does
> not use lastFreeBuffer, it just maintains is, so he removed that as an
> optimization.
Optimiziation / simplification. When building multiple freelists it was
harder to maintain the tail pointer, and since it was never used...
+1 to just applying that part.
> I don't know how significant is the improvement, but if it's measurable we
> could just do that independently of our patches.
I doubt it's really an improvement in any realistic scenario, but it's also
not a regression in any way, since it's never used...
FWIW, I've started to wonder if we shouldn't just get rid of the freelist
entirely. While clocksweep is perhaps minutely slower in a single thread than
the freelist, clock sweep scales *considerably* better [1]. As it's rather
rare to be bottlenecked on clock sweep speed for a single thread (rather then
IO or memory copy overhead), I think it's worth favoring clock sweep.
Also needing to switch between getting buffers from the freelist and the sweep
makes the code more expensive. I think just having the buffer in the sweep,
with a refcount / usagecount of zero would suffice.
That seems particularly advantageous if we invest energy in making the clock
sweep deal well with NUMA systems, because we don't need have both a NUMA
aware freelist and a NUMA aware clock sweep.
Greetings,
Andres Freund
[1]
A single pg_prewarm of a large relation shows a difference between using the
freelist and not that's around the noise level, whereas 40 parallel
pg_prewarms of seperate relations is over 5x faster when disabling the
freelist.
For the test:
- I modified pg_buffercache_evict_* to put buffers onto the freelist
- Ensured all of shared buffers is allocated by querying
pg_shmem_allocations_numa, as otherwise the workload is dominated by the
kernel zeroing out buffers
- used shared_buffers bigger than the data
- data for single threaded is 9.7GB, data for the parallel case is 40
relations of 610MB each.
- in the single threaded case I pinned postgres to a single core, to make sure
core-to-core variation doesn't play a role
- single threaded case
c=1 && psql -Xq -c "select pg_buffercache_evict_all()" -c 'SELECT numa_node, sum(size), count(*) FROM pg_shmem_allocations_numa WHERE size != 0 GROUP BY numa_node;' && pgbench -n -P1 -c$c -j$c -f <(echo "SELECT pg_prewarm('copytest_large');") -t1
concurrent case:
c=40 && psql -Xq -c "select pg_buffercache_evict_all()" -c 'SELECT numa_node, sum(size), count(*) FROM pg_shmem_allocations_numa WHERE size != 0 GROUP BY numa_node;' && pgbench -n -P1 -c$c -j$c -f <(echo "SELECT pg_prewarm('copytest_:client_id');") -t1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 16:55 Greg Burd <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Greg Burd @ 2025-07-09 16:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On Jul 9 2025, at 12:35 pm, Andres Freund <[email protected]> wrote:
> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
> entirely. While clocksweep is perhaps minutely slower in a single
> thread than
> the freelist, clock sweep scales *considerably* better [1]. As it's rather
> rare to be bottlenecked on clock sweep speed for a single thread
> (rather then
> IO or memory copy overhead), I think it's worth favoring clock sweep.
Hey Andres, thanks for spending time on this. I've worked before on
freelist implementations (last one in LMDB) and I think you're onto
something. I think it's an innovative idea and that the speed
difference will either be lost in the noise or potentially entirely
mitigated by avoiding duplicate work.
> Also needing to switch between getting buffers from the freelist and
> the sweep
> makes the code more expensive. I think just having the buffer in the sweep,
> with a refcount / usagecount of zero would suffice.
If you're not already coding this, I'll jump in. :)
> That seems particularly advantageous if we invest energy in making the clock
> sweep deal well with NUMA systems, because we don't need have both a NUMA
> aware freelist and a NUMA aware clock sweep.
100% agree here, very clever approach adapting clock sweep to a NUMA world.
best.
-greg
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 17:13 Andres Freund <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-09 17:13 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-09 12:04:00 +0200, Jakub Wartak wrote:
> On Tue, Jul 8, 2025 at 2:56 PM Andres Freund <[email protected]> wrote:
> > On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> > > On 7/8/25 05:04, Andres Freund wrote:
> > > > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > > > The reason it would be advantageous to put something like the procarray onto
> > > > smaller pages is that otherwise the entire procarray (unless particularly
> > > > large) ends up on a single NUMA node, increasing the latency for backends on
> > > > every other numa node and increasing memory traffic on that node.
> > > >
>
> Sure thing, I fully understand the motivation and underlying reason
> (without claiming that I understand the exact memory access patterns
> that involve procarray/PGPROC/etc and hotspots involved from PG side).
> Any single-liner pgbench help for how to really easily stress the
> PGPROC or procarray?
Unfortunately it's probably going to be slightly more complicated workloads
that show the effect - the very simplest cases don't go iterate through the
procarray itself anymore.
> > > That's why the patch series splits the procarray into multiple pieces,
> > > so that it can be properly distributed on multiple NUMA nodes even with
> > > huge pages. It requires adjusting a couple places accessing the entries,
> > > but it surprised me how limited the impact was.
>
> Yes, and we are discussing if it is worth getting into smaller pages
> for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
> what more even more waste 1GB hugetlb if we dont request 2MB for some
> small structs: btw, we have ability to select MAP_HUGE_2MB vs
> MAP_HUGE_1GB). I'm thinking about two problems:
> - 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning
I'm not really bought into this being a problem. If your system has enough
pressure to swap out the PGPROC array, you're so hosed that this won't make a
difference.
> - using libnuma often leads to MPOL_BIND which disarms NUMA
> autobalancing, BUT apparently there are set_mempolicy(2)/mbind(2) and
> since 5.12+ kernel they can take additional flag
> MPOL_F_NUMA_BALANCING(!), so this looks like it has potential to move
> memory anyway (if way too many tasks are relocated, so would be
> memory?). It is available only in recent libnuma as
> numa_set_membind_balancing(3), but sadly there's no way via libnuma to
> do mbind(MPOL_F_NUMA_BALANCING) for a specific addr only? I mean it
> would have be something like MPOL_F_NUMA_BALANCING | MPOL_PREFERRED?
> (select one node from many for each node while still allowing
> balancing?), but in [1][2] (2024) it is stated that "It's not
> legitimate (yet) to use MPOL_PREFERRED + MPOL_F_NUMA_BALANCING.", but
> maybe stuff has been improved since then.
>
> Something like:
> PGPROC/procarray 2MB page for node#1 - mbind(addr1,
> MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [0,1]);
> PGPROC/procarray 2MB page for node#2 - mbind(addr2,
> MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [1,0]);
I'm rather doubtful that it's a good idea to combine numa awareness with numa
balancing. Numa balancing adds latency and makes it much more expensive for
userspace to act in a numa aware way, since it needs to regularly update its
knowledge about where memory resides.
> > Sure, you can do that, but it does mean that iterations over the procarray now
> > have an added level of indirection...
>
> So the most efficient would be the old-way (no indirections) vs
> NUMA-way? Can this be done without #ifdefs at all?
If we used 4k pages for the procarray we would just have ~4 procs on one page,
if that range were marked as interleaved, it'd probably suffice.
> > > The thing I'm not sure about is how much this actually helps with the
> > > traffic between node. Sure, if we pick a PGPROC from the same node, and
> > > the task does not get moved, it'll be local traffic. But if the task
> > > moves, there'll be traffic.
>
> With MPOL_F_NUMA_BALANCING, that should "auto-tune" in the worst case?
I doubt that NUMA balancing is going to help a whole lot here, there are too
many procs on one page for that to be helpful. One thing that might be worth
doing is to *increase* the size of PGPROC by moving other pieces of data that
are keyed by ProcNumber into PGPROC.
I think the main thing to avoid is the case where all of PGPROC, buffer
mapping table, ... resides on one NUMA node (e.g. because it's the one
postmaster was scheduled on), as the increased memory traffic will lead to
queries on that node being slower than the other node.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 17:23 Andres Freund <[email protected]>
parent: Greg Burd <[email protected]>
0 siblings, 2 replies; 89+ messages in thread
From: Andres Freund @ 2025-07-09 17:23 UTC (permalink / raw)
To: Greg Burd <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
> On Jul 9 2025, at 12:35 pm, Andres Freund <[email protected]> wrote:
>
> > FWIW, I've started to wonder if we shouldn't just get rid of the freelist
> > entirely. While clocksweep is perhaps minutely slower in a single
> > thread than
> > the freelist, clock sweep scales *considerably* better [1]. As it's rather
> > rare to be bottlenecked on clock sweep speed for a single thread
> > (rather then
> > IO or memory copy overhead), I think it's worth favoring clock sweep.
>
> Hey Andres, thanks for spending time on this. I've worked before on
> freelist implementations (last one in LMDB) and I think you're onto
> something. I think it's an innovative idea and that the speed
> difference will either be lost in the noise or potentially entirely
> mitigated by avoiding duplicate work.
Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
perform better because it doesn't need to maintain the freelist anymore...
> > Also needing to switch between getting buffers from the freelist and
> > the sweep
> > makes the code more expensive. I think just having the buffer in the sweep,
> > with a refcount / usagecount of zero would suffice.
>
> If you're not already coding this, I'll jump in. :)
My experimental patch is literally a four character addition ;), namely adding
"0 &&" to the relevant code in StrategyGetBuffer().
Obviously a real patch would need to do some more work than that. Feel free
to take on that project, I am not planning on tackling that in near term.
There's other things around this that could use some attention. It's not hard
to see clock sweep be a bottleneck in concurrent workloads - partially due to
the shared maintenance of the clock hand. A NUMAed clock sweep would address
that. However, we also maintain StrategyControl->numBufferAllocs, which is a
significant contention point and would not necessarily be removed by a
NUMAificiation of the clock sweep.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-09 17:57 Andres Freund <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Andres Freund @ 2025-07-09 17:57 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-08 16:06:00 +0000, Cédric Villemain wrote:
> > Assuming we want to actually pin tasks from within Postgres, what I
> > think might work is allowing modules to "advise" on where to place the
> > task. But the decision would still be done by core.
>
> Possibly exactly what you're doing in proc.c when managing allocation of
> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
> candidates), I didn't get that they require information not available to any
> process executing code from a module.
> Parts of your code where you assign/define policy could be in one or more
> relevant routines of a "numa profile manager", like in an
> initProcessRoutine(), and registered in pmroutine struct:
>
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
> pmroutine->init_process != NULL)
> pmroutine->init_process(MyProc);
>
> This way it's easier to manage alternative policies, and also to be able to
> adjust when hardware and linux kernel changes.
I am doubtful this makes sense - as you can see patch 05 needs to change a
fair bit of core code to make this work, there's no way we can delegate much
of that to an extension.
But even if it's doable, I think it's *very* premature to focus on such
extensibility at this point - we need to get the basics into a mergeable
state, if you then want to argue for adding extensibility, we can do that at
this stage. Trying to design this for extensibility from the get go, where
that extensibility is very unlikely to be used widely, seems rather likely to
just tank this entire project without getting us anything in return.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-09 19:42 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
3 siblings, 2 replies; 89+ messages in thread
From: Andres Freund @ 2025-07-09 19:42 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
Thanks for working on this! I think it's an area we have long neglected...
On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> Each patch has a numa_ GUC, intended to enable/disable that part. This
> is meant to make development easier, not as a final interface. I'm not
> sure how exactly that should look. It's possible some combinations of
> GUCs won't work, etc.
Wonder if some of it might be worth putting into a multi-valued GUC (like
debug_io_direct).
> 1) v1-0001-NUMA-interleaving-buffers.patch
>
> This is the main thing when people think about NUMA - making sure the
> shared buffers are allocated evenly on all the nodes, not just on a
> single node (which can happen easily with warmup). The regular memory
> interleaving would address this, but it also has some disadvantages.
>
> Firstly, it's oblivious to the contents of the shared memory segment,
> and we may not want to interleave everything. It's also oblivious to
> alignment of the items (a buffer can easily end up "split" on multiple
> NUMA nodes), or relationship between different parts (e.g. there's a
> BufferBlock and a related BufferDescriptor, and those might again end up
> on different nodes).
Two more disadvantages:
With OS interleaving postgres doesn't (not easily at least) know about what
maps to what, which means postgres can't do stuff like numa aware buffer
replacement.
With OS interleaving the interleaving is "too fine grained", with pages being
mapped at each page boundary, making it less likely for things like one
strategy ringbuffer to reside on a single numa node.
I wonder if we should *increase* the size of shared_buffers whenever huge
pages are in use and there's padding space due to the huge page
boundaries. Pretty pointless to waste that memory if we can instead use if for
the buffer pool. Not that big a deal with 2MB huge pages, but with 1GB huge
pages...
> 4) v1-0004-NUMA-partition-buffer-freelist.patch
>
> Right now we have a single freelist, and in busy instances that can be
> quite contended. What's worse, the freelist may trash between different
> CPUs, NUMA nodes, etc. So the idea is to have multiple freelists on
> subsets of buffers. The patch implements multiple strategies how the
> list can be split (configured using "numa_partition_freelist" GUC), for
> experimenting:
>
> * node - One list per NUMA node. This is the most natural option,
> because we now know which buffer is on which node, so we can ensure a
> list for a node only has buffers from that list.
>
> * cpu - One list per CPU. Pretty simple, each CPU gets it's own list.
>
> * pid - Similar to "cpu", but the processes are mapped to lists based on
> PID, not CPU ID.
>
> * none - nothing, sigle freelist
>
> Ultimately, I think we'll want to go with "node", simply because it
> aligns with the buffer interleaving. But there are improvements needed.
I think we might eventually want something more granular than just "node" -
the freelist (and the clock sweep) can become a contention point even within
one NUMA node. I'm imagining something like an array of freelists/clocksweep
states, where the current numa node selects a subset of the array and the cpu
is used to choose the entry within that list.
But we can do that later, that should be a fairly simple extension of what
you're doing.
> The other missing part is clocksweep - there's still just a single
> instance of clocksweep, feeding buffers to all the freelists. But that's
> clearly a problem, because the clocksweep returns buffers from all NUMA
> nodes. The clocksweep really needs to be partitioned the same way as a
> freelists, and each partition will operate on a subset of buffers (from
> the right NUMA node).
>
> I do have a separate experimental patch doing something like that, I
> need to make it part of this branch.
I'm really curious about that patch, as I wrote elsewhere in this thread, I
think we should just get rid of the freelist alltogether. Even if we don't do
so, in a steady state system the clock sweep is commonly much more important
than the freelist...
> 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
>
> Another area that seems like it might benefit from NUMA is PGPROC, so I
> gave it a try. It turned out somewhat challenging. Similarly to buffers
> we have two pieces that need to be located in a coordinated way - PGPROC
> entries and fast-path arrays. But we can't use the same approach as for
> buffers/descriptors, because
>
> (a) Neither of those pieces aligns with memory page size (PGPROC is
> ~900B, fast-path arrays are variable length).
> (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> rather high max_connections before we use multiple huge pages.
We should probably pad them regardless? Right now sizeof(PGPROC) happens to be
multiple of 64 (i.e. the most common cache line size), but that hasn't always
been the case, and isn't the case on systems with 128 bit cachelines like
common ARMv8 systems. And having one cacheline hold one backends fast path
states and another backend's xmin doesn't sound like a recipe for good
performance.
Seems like we should also do some reordering of the contents within PGPROC. We
have e.g. have very frequently changing data (->waitStatus, ->lwWaiting) in
the same caceheline as almost immutable data (->pid, ->pgxactoff,
->databaseId,).
> So what I did instead is splitting the whole PGPROC array into one array
> per NUMA node, and one array for auxiliary processes and 2PC xacts. So
> with 4 NUMA nodes there are 5 separate arrays, for example. Each array
> is a multiple of memory pages, so we may waste some of the memory. But
> that's simply how NUMA works - page granularity.
Theoretically we could use the "padding" memory at the end of each NUMA node's
PGPROC array to for the 2PC entries, for those we presumably don't care for
locality. Not sure it's worth the complexity though.
For a while I thought I had a better solution: Given that we're going to waste
all the "padding" memory, why not just oversize the PGPROC array so that it
spans the required number of NUMA nodes?
The problem is that that would lead to ProcNumbers to get much larger, and we
do have other arrays that are keyed by ProcNumber. Which probably makes this
not so great an idea.
> This however makes one particular thing harder - in a couple places we
> accessed PGPROC entries through PROC_HDR->allProcs, which was pretty
> much just one large array. And GetNumberFromPGProc() relied on array
> arithmetics to determine procnumber. With the array partitioned, this
> can't work the same way.
>
> But there's a simple solution - if we turn allProcs into an array of
> *pointers* to PGPROC arrays, there's no issue. All the places need a
> pointer anyway. And then we need an explicit procnumber field in PGPROC,
> instead of calculating it.
>
> There's a chance this have negative impact on code that accessed PGPROC
> very often, but so far I haven't seen such cases. But if you can come up
> with such examples, I'd like to see those.
I'd not be surprised if there were overhead, adding a level of indirection to
things like ProcArrayGroupClearXid(), GetVirtualXIDsDelayingChkpt(),
SignalVirtualTransaction() probably won't be free.
BUT: For at least some of these a better answer might be to add additional
"dense" arrays like we have for xids etc, so they don't need to trawl through
PGPROCs.
> There's another detail - when obtaining a PGPROC entry in InitProcess(),
> we try to get an entry from the same NUMA node. And only if that doesn't
> work, we grab the first one from the list (there's still just one PGPROC
> freelist, I haven't split that - maybe we should?).
I guess it might be worth partitioning the freelist, iterating through a few
thousand links just to discover that there's no free proc on the current numa
node, while holding a spinlock, doesn't sound great. Even if it's likely
rarely a huge issue compared to other costs.
> The other thing I haven't thought about very much is determining on
> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> simply inherit/determine that at the start through libnuma, not through
> some custom PG configuration (which the patch [2] proposed to do).
That seems like the right thing to me.
One thing that this patchset afaict doesn't address so far is that there is a
fair bit of other important shared memory that this patch doesn't set up
intelligently e.g. the buffer mapping table itself (but there are loads of
other cases). Because we touch a lot of that memory during startup, most it
will be allocated on whatever NUMA node postmaster was scheduled. I suspect
that the best we can do for parts of shared memory where we don't have
explicit NUMA awareness is to default to an interleave policy.
> From 9712e50d6d15c18ea2c5fcf457972486b0d4ef53 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <[email protected]>
> Date: Tue, 6 May 2025 21:12:21 +0200
> Subject: [PATCH v1 1/6] NUMA: interleaving buffers
>
> Ensure shared buffers are allocated from all NUMA nodes, in a balanced
> way, instead of just using the node where Postgres initially starts, or
> where the kernel decides to migrate the page, etc. With pre-warming
> performed by a single backend, this can easily result in severely
> unbalanced memory distribution (with most from a single NUMA node).
>
> The kernel would eventually move some of the memory to other nodes
> (thanks to zone_reclaim), but that tends to take a long time. So this
> patch improves predictability, reduces the time needed for warmup
> during benchmarking, etc. It's less dependent on what the CPU
> scheduler does, etc.
FWIW, I don't think zone_reclaim_mode will commonly do that? Even if enabled,
which I don't think it is anymore by default. At least huge pages can't be
reclaimed by the kernel, but even when not using huge pages, I think the only
scenario where that would happen is if shared_buffers were swapped out.
Numa balancing might eventually "fix" such an imbalance though.
> diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
> index ed1dc488a42..2ad34624c49 100644
> --- a/src/backend/storage/buffer/buf_init.c
> +++ b/src/backend/storage/buffer/buf_init.c
> @@ -14,9 +14,17 @@
> */
> #include "postgres.h"
>
> +#ifdef USE_LIBNUMA
> +#include <numa.h>
> +#include <numaif.h>
> +#endif
> +
I wonder how much of this we should try to put into port/pg_numa.c. Having
direct calls to libnuma code all over the backend will make it rather hard to
add numa awareness for hypothetical platforms not using libnuma compatible
interfaces.
> +/* number of buffers allocated on the same NUMA node */
> +static int64 numa_chunk_buffers = -1;
Given that NBuffers is a 32bit quantity, this probably doesn't need to be
64bit... Anyway, I'm not going to review on that level going forward, the
patch is probably in too early a state for that.
> @@ -71,18 +92,80 @@ BufferManagerShmemInit(void)
> foundDescs,
> foundIOCV,
> foundBufCkpt;
> + Size mem_page_size;
> + Size buffer_align;
> +
> + /*
> + * XXX A bit weird. Do we need to worry about postmaster? Could this even
> + * run outside postmaster? I don't think so.
It can run in single user mode - but that shouldn't prevent us from using
pg_get_shmem_pagesize().
> + * XXX Another issue is we may get different values than when sizing the
> + * the memory, because at that point we didn't know if we get huge pages,
> + * so we assumed we will. Shouldn't cause crashes, but we might allocate
> + * shared memory and then not use some of it (because of the alignment
> + * that we don't actually need). Not sure about better way, good for now.
> + */
Ugh, not seeing a great way to deal with that either.
> + * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
> + * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
> + * that doesn't seem quite worth it. Maybe we should simply align to
> + * BLCKSZ, so that buffers don't get split? Still, we might interfere with
> + * other stuff stored in shared memory that we want to allocate on a
> + * particular NUMA node (e.g. ProcArray).
> + *
> + * XXX Maybe with "too large" huge pages we should just not do this, or
> + * maybe do this only for sufficiently large areas (e.g. shared buffers,
> + * but not ProcArray).
I think that's right - there's no point in using 1GB pages for anything other
than shared_buffers, we should allocate shared_buffers separately.
> +/*
> + * Determine the size of memory page.
> + *
> + * XXX This is a bit tricky, because the result depends at which point we call
> + * this. Before the allocation we don't know if we succeed in allocating huge
> + * pages - but we have to size everything for the chance that we will. And then
> + * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
> + * pages. But at that point we can't adjust the sizing.
> + *
> + * XXX Maybe with huge_pages=try we should do the sizing twice - first with
> + * huge pages, and if that fails, then without them. But not for this patch.
> + * Up to this point there was no such dependency on huge pages.
Doing it twice sounds somewhat nasty - but perhaps we could just have the
shmem size infrastructure compute two different numbers, one for use with huge
pages and one without?
> +static int64
> +choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes)
> +{
> + int64 num_items;
> + int64 max_items;
> +
> + /* make sure the chunks will align nicely */
> + Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
> + Assert(mem_page_size % sizeof(BufferDescPadded) == 0);
> + Assert(((BLCKSZ % mem_page_size) == 0) || ((mem_page_size % BLCKSZ) == 0));
> +
> + /*
> + * The minimum number of items to fill a memory page with descriptors and
> + * blocks. The NUMA allocates memory in pages, and we need to do that for
> + * both buffers and descriptors.
> + *
> + * In practice the BLCKSZ doesn't really matter, because it's much larger
> + * than BufferDescPadded, so the result is determined buffer descriptors.
> + * But it's clearer this way.
> + */
> + num_items = Max(mem_page_size / sizeof(BufferDescPadded),
> + mem_page_size / BLCKSZ);
> +
> + /*
> + * We shouldn't use chunks larger than NBuffers/num_nodes, because with
> + * larger chunks the last NUMA node would end up with much less memory (or
> + * no memory at all).
> + */
> + max_items = (NBuffers / num_nodes);
> +
> + /*
> + * Did we already exceed the maximum desirable chunk size? That is, will
> + * the last node get less than one whole chunk (or no memory at all)?
> + */
> + if (num_items > max_items)
> + elog(WARNING, "choose_chunk_buffers: chunk items exceeds max (%ld > %ld)",
> + num_items, max_items);
> +
> + /* grow the chunk size until we hit the max limit. */
> + while (2 * num_items <= max_items)
> + num_items *= 2;
Something around this logic leads to a fair bit of imbalance - I started postgres with
huge_page_size=1GB, shared_buffers=4GB on a 2 node system and that results in
postgres[4188255][1]=# SELECT * FROM pg_shmem_allocations_numa WHERE name in ('Buffer Blocks', 'Buffer Descriptors');
┌────────────────────┬───────────┬────────────┐
│ name │ numa_node │ size │
├────────────────────┼───────────┼────────────┤
│ Buffer Blocks │ 0 │ 5368709120 │
│ Buffer Blocks │ 1 │ 1073741824 │
│ Buffer Descriptors │ 0 │ 1073741824 │
│ Buffer Descriptors │ 1 │ 1073741824 │
└────────────────────┴───────────┴────────────┘
(4 rows)
With shared_buffers=8GB postgres failed to start, even though 16 1GB huge
pages are available, as 18GB were requested.
After increasing the limit, the top allocations were as follows:
postgres[4189384][1]=# SELECT * FROM pg_shmem_allocations ORDER BY allocated_size DESC LIMIT 5;
┌──────────────────────┬─────────────┬────────────┬────────────────┐
│ name │ off │ size │ allocated_size │
├──────────────────────┼─────────────┼────────────┼────────────────┤
│ Buffer Blocks │ 1192223104 │ 9663676416 │ 9663676416 │
│ PGPROC structures │ 10970279808 │ 3221733342 │ 3221733376 │
│ Fast-Path Lock Array │ 14192013184 │ 3221396544 │ 3221396608 │
│ Buffer Descriptors │ 51372416 │ 1140850688 │ 1140850688 │
│ (null) │ 17468590976 │ 785020032 │ 785020032 │
└──────────────────────┴─────────────┴────────────┴────────────────┘
With a fair bit of imbalance:
postgres[4189384][1]=# SELECT * FROM pg_shmem_allocations_numa WHERE name in ('Buffer Blocks', 'Buffer Descriptors');
┌────────────────────┬───────────┬────────────┐
│ name │ numa_node │ size │
├────────────────────┼───────────┼────────────┤
│ Buffer Blocks │ 0 │ 8589934592 │
│ Buffer Blocks │ 1 │ 2147483648 │
│ Buffer Descriptors │ 0 │ 0 │
│ Buffer Descriptors │ 1 │ 2147483648 │
└────────────────────┴───────────┴────────────┘
(4 rows)
Note that the buffer descriptors are all on node 1.
> +/*
> + * Calculate the NUMA node for a given buffer.
> + */
> +int
> +BufferGetNode(Buffer buffer)
> +{
> + /* not NUMA interleaving */
> + if (numa_chunk_buffers == -1)
> + return -1;
> +
> + return (buffer / numa_chunk_buffers) % numa_nodes;
> +}
FWIW, this is likely rather expensive - when not a compile time constant,
divisions and modulo can take a fair number of cycles.
> +/*
> + * pg_numa_interleave_memory
> + * move memory to different NUMA nodes in larger chunks
> + *
> + * startptr - start of the region (should be aligned to page size)
> + * endptr - end of the region (doesn't need to be aligned)
> + * mem_page_size - size of the memory page size
> + * chunk_size - size of the chunk to move to a single node (should be multiple
> + * of page size
> + * num_nodes - number of nodes to allocate memory to
> + *
> + * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
> + * That might be more efficient than numa_move_pages, as it works on larger
> + * chunks of memory, not individual system pages, I think.
> + *
> + * XXX The "interleave" name is not quite accurate, I guess.
> + */
> +static void
> +pg_numa_interleave_memory(char *startptr, char *endptr,
> + Size mem_page_size, Size chunk_size,
> + int num_nodes)
> +{
Seems like this should be in pg_numa.c?
> diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
> index 69b6a877dc9..c07de903f76 100644
> --- a/src/bin/pgbench/pgbench.c
> +++ b/src/bin/pgbench/pgbench.c
I assume those changes weren't intentionally part of this patch...
> From 6505848ac8359c8c76dfbffc7150b6601ab07601 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <[email protected]>
> Date: Thu, 22 May 2025 18:38:41 +0200
> Subject: [PATCH v1 4/6] NUMA: partition buffer freelist
>
> Instead of a single buffer freelist, partition into multiple smaller
> lists, to reduce lock contention, and to spread the buffers over all
> NUMA nodes more evenly.
>
> There are four strategies, specified by GUC numa_partition_freelist
>
> * none - single long freelist, should work just like now
>
> * node - one freelist per NUMA node, with only buffers from that node
>
> * cpu - one freelist per CPU
>
> * pid - freelist determined by PID (same number of freelists as 'cpu')
>
> When allocating a buffer, it's taken from the correct freelist (e.g.
> same NUMA node).
>
> Note: This is (probably) more important than partitioning ProcArray.
>
> +/*
> + * Represents one freelist partition.
> + */
> +typedef struct BufferStrategyFreelist
> +{
> + /* Spinlock: protects the values below */
> + slock_t freelist_lock;
> +
> + /*
> + * XXX Not sure why this needs to be aligned like this. Need to ask
> + * Andres.
> + */
> + int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
> + * unused buffers */
> +
> + /* Number of buffers consumed from this list. */
> + uint64 consumed;
> +} BufferStrategyFreelist;
I think this might be a leftover from measuring performance of a *non*
partitioned freelist. I saw unnecessar contention between
BufferStrategyControl->{nextVictimBuffer,buffer_strategy_lock,numBufferAllocs}
and was testing what effect the simplest avoidance scheme has.
I don't this should be part of this patchset.
>
> /*
> * The shared freelist control information.
> @@ -39,8 +66,6 @@ typedef struct
> */
> pg_atomic_uint32 nextVictimBuffer;
>
> - int firstFreeBuffer; /* Head of list of unused buffers */
> -
> /*
> * Statistics. These counters should be wide enough that they can't
> * overflow during a single bgwriter cycle.
> @@ -51,13 +76,27 @@ typedef struct
> /*
> * Bgworker process to be notified upon activity or -1 if none. See
> * StrategyNotifyBgWriter.
> + *
> + * XXX Not sure why this needs to be aligned like this. Need to ask
> + * Andres. Also, shouldn't the alignment be specified after, like for
> + * "consumed"?
> */
> - int bgwprocno;
> + int __attribute__((aligned(64))) bgwprocno;
> +
> + BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
> } BufferStrategyControl;
Here the reason was that it's silly to put almost-readonly data (like
bgwprocno) onto the same cacheline as very frequently modified data like
->numBufferAllocs. That causes unnecessary cache misses in many
StrategyGetBuffer() calls, as another backend's StrategyGetBuffer() will
always have modified ->numBufferAllocs and either ->buffer_strategy_lock or
->nextVictimBuffer.
>
> +static BufferStrategyFreelist *
> +ChooseFreeList(void)
> +{
> + unsigned cpu;
> + unsigned node;
> + int rc;
> +
> + int freelist_idx;
> +
> + /* freelist not partitioned, return the first (and only) freelist */
> + if (numa_partition_freelist == FREELIST_PARTITION_NONE)
> + return &StrategyControl->freelists[0];
> +
> + /*
> + * freelist is partitioned, so determine the CPU/NUMA node, and pick a
> + * list based on that.
> + */
> + rc = getcpu(&cpu, &node);
> + if (rc != 0)
> + elog(ERROR, "getcpu failed: %m");
Probably should put this into somewhere abstracted away...
> + /*
> + * Pick the freelist, based on CPU, NUMA node or process PID. This matches
> + * how we built the freelists above.
> + *
> + * XXX Can we rely on some of the values (especially strategy_nnodes) to
> + * be a power-of-2? Then we could replace the modulo with a mask, which is
> + * likely more efficient.
> + */
> + switch (numa_partition_freelist)
> + {
> + case FREELIST_PARTITION_CPU:
> + freelist_idx = cpu % strategy_ncpus;
As mentioned earlier, modulo is rather expensive for something executed so
frequently...
> + break;
> +
> + case FREELIST_PARTITION_NODE:
> + freelist_idx = node % strategy_nnodes;
> + break;
Here we shouldn't need modulo, right?
> +
> + case FREELIST_PARTITION_PID:
> + freelist_idx = MyProcPid % strategy_ncpus;
> + break;
> +
> + default:
> + elog(ERROR, "unknown freelist partitioning value");
> + }
> +
> + return &StrategyControl->freelists[freelist_idx];
> +}
> /* size of lookup hash table ... see comment in StrategyInitialize */
> size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
>
> /* size of the shared replacement strategy control block */
> - size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
> + size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
> +
> + /*
> + * Allocate one frelist per CPU. We might use per-node freelists, but the
> + * assumption is the number of CPUs is less than number of NUMA nodes.
> + *
> + * FIXME This assumes the we have more CPUs than NUMA nodes, which seems
> + * like a safe assumption. But maybe we should calculate how many elements
> + * we actually need, depending on the GUC? Not a huge amount of memory.
FWIW, I don't think that's a safe assumption anymore. With CXL we can get a)
PCIe attached memory and b) remote memory as a separate NUMA nodes, and that
very well could end up as more NUMA nodes than cores.
Ugh, -ETOOLONG. Gotta schedule some other things...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-10 10:15 Jakub Wartak <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Jakub Wartak @ 2025-07-10 10:15 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jul 9, 2025 at 7:13 PM Andres Freund <[email protected]> wrote:
> > Yes, and we are discussing if it is worth getting into smaller pages
> > for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
> > what more even more waste 1GB hugetlb if we dont request 2MB for some
> > small structs: btw, we have ability to select MAP_HUGE_2MB vs
> > MAP_HUGE_1GB). I'm thinking about two problems:
> > - 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning
>
> I'm not really bought into this being a problem. If your system has enough
> pressure to swap out the PGPROC array, you're so hosed that this won't make a
> difference.
OK I need to bend here, yet still part of me believes that the
situation where we have hugepages (for 'Buffer Blocks') and yet some
smaller more, but way critical structs are more likely to be swapped
out due to pressure of some backend-gone-wild random mallocs() is
unhealthy to me (especially the fact the OS might prefer swapping on
per node rather than global picture)
> I'm rather doubtful that it's a good idea to combine numa awareness with numa
> balancing. Numa balancing adds latency and makes it much more expensive for
> userspace to act in a numa aware way, since it needs to regularly update its
> knowledge about where memory resides.
Well the problem is that backends come here and go to random CPUs
often (migrated++ on very high backend counts and non-uniform
workloads in terms of backend-CPU usage), but the autobalancing
doesn't need to be on or off for everything. It could be autobalancing
for a certain memory region and it is not affecting the app in any way
(well, other than those minor page faulting, literally ).
> If we used 4k pages for the procarray we would just have ~4 procs on one page,
> if that range were marked as interleaved, it'd probably suffice.
OK, this sounds like the best and simplest proposal to me, yet the
patch doesn't do OS-based interleaving for those today. Gonna try that
mlock() sooner or later... ;)
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-10 10:16 Jakub Wartak <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Jakub Wartak @ 2025-07-10 10:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jul 9, 2025 at 9:42 PM Andres Freund <[email protected]> wrote:
> On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> > Each patch has a numa_ GUC, intended to enable/disable that part. This
> > is meant to make development easier, not as a final interface. I'm not
> > sure how exactly that should look. It's possible some combinations of
> > GUCs won't work, etc.
>
> Wonder if some of it might be worth putting into a multi-valued GUC (like
> debug_io_direct).
Long-term or for experimentation? Also please see below as it is related:
[..]
> FWIW, I don't think that's a safe assumption anymore. With CXL we can get a)
> PCIe attached memory and b) remote memory as a separate NUMA nodes, and that
> very well could end up as more NUMA nodes than cores.
In my earlier apparently very way too naive approach, I've tried to
handle this CXL scenario, but I'm afraid this cannot be done without
further configuration, please see review/use cases [1] and [2]
-J.
[1] https://www.postgresql.org/message-id/attachment/178119/v4-0001-Add-capability-to-interleave-shared-...
- just see sgml/GUC and we have numa_parse_nodestring(3)
[2] https://www.postgresql.org/message-id/aAKPMrX1Uq6quKJy%40ip-10-97-1-34.eu-west-3.compute.internal
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-10 12:13 Burd, Greg <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Burd, Greg @ 2025-07-10 12:13 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
> On Jul 9, 2025, at 1:23 PM, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>> On Jul 9 2025, at 12:35 pm, Andres Freund <[email protected]> wrote:
>>
>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>> entirely. While clocksweep is perhaps minutely slower in a single
>>> thread than
>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>> rare to be bottlenecked on clock sweep speed for a single thread
>>> (rather then
>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>>
>> Hey Andres, thanks for spending time on this. I've worked before on
>> freelist implementations (last one in LMDB) and I think you're onto
>> something. I think it's an innovative idea and that the speed
>> difference will either be lost in the noise or potentially entirely
>> mitigated by avoiding duplicate work.
>
> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
> perform better because it doesn't need to maintain the freelist anymore...
>
>
>>> Also needing to switch between getting buffers from the freelist and
>>> the sweep
>>> makes the code more expensive. I think just having the buffer in the sweep,
>>> with a refcount / usagecount of zero would suffice.
>>
>> If you're not already coding this, I'll jump in. :)
>
> My experimental patch is literally a four character addition ;), namely adding
> "0 &&" to the relevant code in StrategyGetBuffer().
>
> Obviously a real patch would need to do some more work than that. Feel free
> to take on that project, I am not planning on tackling that in near term.
>
I started on this last night, making good progress. Thanks for the inspiration. I'll create a new thread to track the work and cross-reference when I have something reasonable to show (hopefully later today).
> There's other things around this that could use some attention. It's not hard
> to see clock sweep be a bottleneck in concurrent workloads - partially due to
> the shared maintenance of the clock hand. A NUMAed clock sweep would address
> that.
Working on it. Other than NUMA-fying clocksweep there is a function have_free_buffer() that might be a tad tricky to re-implement efficiently and/or make NUMA aware. Or maybe I can remove that too? It is used in autoprewarm.c and possibly other extensions, but no where else in core.
> However, we also maintain StrategyControl->numBufferAllocs, which is a
> significant contention point and would not necessarily be removed by a
> NUMAificiation of the clock sweep.
Yep, I noted this counter and its potential for contention too. Fortunately, it seems like it is only used so that "bgwriter can estimate the rate of buffer consumption" which to me opens the door to a less accurate partitioned counter, perhaps something lock-free (no mutex/CAS) that is bucketed then combined when read.
A quick look at bufmgr.c indicates that recent_allocs (which is StrategyControl->numBufferAllocs) is used to track a "moving average" and other voodoo there I've yet to fully grok. Any thoughts on this approximate count approach?
Also, what are your thoughts on updating the algorithm to CLOCK-Pro [1] while I'm there? I guess I'd have to try it out, measure it a lot and see if there are any material benefits. Maybe I'll keep that for a future patch, or at least layer it... back to work!
> Greetings,
>
> Andres Freund
best.
-greg
[1] https://www.usenix.org/legacy/publications/library/proceedings/usenix05/tech/general/full_papers/jia...
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-10 14:17 Bertrand Drouvot <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Bertrand Drouvot @ 2025-07-10 14:17 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Wed, Jul 09, 2025 at 03:42:26PM -0400, Andres Freund wrote:
> Hi,
>
> Thanks for working on this!
Indeed, thanks!
> On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> > 1) v1-0001-NUMA-interleaving-buffers.patch
> >
> > This is the main thing when people think about NUMA - making sure the
> > shared buffers are allocated evenly on all the nodes, not just on a
> > single node (which can happen easily with warmup). The regular memory
> > interleaving would address this, but it also has some disadvantages.
> >
> > Firstly, it's oblivious to the contents of the shared memory segment,
> > and we may not want to interleave everything. It's also oblivious to
> > alignment of the items (a buffer can easily end up "split" on multiple
> > NUMA nodes), or relationship between different parts (e.g. there's a
> > BufferBlock and a related BufferDescriptor, and those might again end up
> > on different nodes).
>
> Two more disadvantages:
>
> With OS interleaving postgres doesn't (not easily at least) know about what
> maps to what, which means postgres can't do stuff like numa aware buffer
> replacement.
>
> With OS interleaving the interleaving is "too fine grained", with pages being
> mapped at each page boundary, making it less likely for things like one
> strategy ringbuffer to reside on a single numa node.
> > There's a secondary benefit of explicitly assigning buffers to nodes,
> > using this simple scheme - it allows quickly determining the node ID
> > given a buffer ID. This is helpful later, when building freelist.
I do think this is a big advantage as compare to the OS interleaving.
> I wonder if we should *increase* the size of shared_buffers whenever huge
> pages are in use and there's padding space due to the huge page
> boundaries. Pretty pointless to waste that memory if we can instead use if for
> the buffer pool. Not that big a deal with 2MB huge pages, but with 1GB huge
> pages...
I think that makes sense, except maybe for operations that need to scan
the whole buffer pool (i.e related to BUF_DROP_FULL_SCAN_THRESHOLD)?
> > 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
> >
> > Another area that seems like it might benefit from NUMA is PGPROC, so I
> > gave it a try. It turned out somewhat challenging. Similarly to buffers
> > we have two pieces that need to be located in a coordinated way - PGPROC
> > entries and fast-path arrays. But we can't use the same approach as for
> > buffers/descriptors, because
> >
> > (a) Neither of those pieces aligns with memory page size (PGPROC is
> > ~900B, fast-path arrays are variable length).
>
> > (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> > rather high max_connections before we use multiple huge pages.
>
> Right now sizeof(PGPROC) happens to be multiple of 64 (i.e. the most common
> cache line size)
Oh right, it's currently 832 bytes and the patch extends that to 840 bytes.
With a bit of reordering:
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 5cb1632718e..2ed2f94202a 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -194,8 +194,6 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
- int procnumber; /* index in ProcGlobal->allProcs */
-
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -243,6 +241,7 @@ struct PGPROC
/* Support for condition variables. */
proclist_node cvWaitLink; /* position in CV wait list */
+ int procnumber; /* index in ProcGlobal->allProcs */
/* Info about lock the process is currently waiting for, if any. */
/* waitLock and waitProcLock are NULL if not currently waiting. */
@@ -268,6 +267,7 @@ struct PGPROC
*/
XLogRecPtr waitLSN; /* waiting for this LSN or higher */
int syncRepState; /* wait state for sync rep */
+ int numa_node;
dlist_node syncRepLinks; /* list link if process is in syncrep queue */
/*
@@ -321,9 +321,6 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
-
- /* NUMA node */
- int numa_node;
};
That could be back to 832 (the order does not make sense logically speaking
though).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach
@ 2025-07-10 15:20 Tomas Vondra <[email protected]>
parent: Cédric Villemain <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-07-10 15:20 UTC (permalink / raw)
To: Cédric Villemain <[email protected]>; Andres Freund <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/9/25 08:40, Cédric Villemain wrote:
>> On 7/8/25 18:06, Cédric Villemain wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On 7/8/25 03:55, Cédric Villemain wrote:
>>>>> Hi Andres,
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>>>> to the
>>>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>>>> integrating
>>>>>>> NUMA-specific management directly into core PostgreSQL in such a
>>>>>>> way.
>>>>>>
>>>>>> I think it's actually the opposite - whenever we pushed stuff like
>>>>>> this
>>>>>> outside of core it has hurt postgres substantially. Not having
>>>>>> replication in
>>>>>> core was a huge mistake. Not having HA management in core is
>>>>>> probably the
>>>>>> biggest current adoption hurdle for postgres.
>>>>>>
>>>>>> To deal better with NUMA we need to improve memory placement and
>>>>>> various
>>>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>>>> to do
>>>>>> outside of core.
>>>>>
>>>>> Except the backend pinning which is easy to achieve, thus my
>>>>> comment on
>>>>> the related patch.
>>>>> I'm not claiming NUMA memory and all should be managed outside of core
>>>>> (though I didn't read other patches yet).
>>>>>
>>>>
>>>> But an "optimal backend placement" seems to very much depend on
>>>> where we
>>>> placed the various pieces of shared memory. Which the external module
>>>> will have trouble following, I suspect.
>>>>
>>>> I still don't have any idea what exactly would the external module do,
>>>> how would it decide where to place the backend. Can you describe some
>>>> use case with an example?
>>>>
>>>> Assuming we want to actually pin tasks from within Postgres, what I
>>>> think might work is allowing modules to "advise" on where to place the
>>>> task. But the decision would still be done by core.
>>>
>>> Possibly exactly what you're doing in proc.c when managing allocation of
>>> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
>>> candidates), I didn't get that they require information not available to
>>> any process executing code from a module.
>>>
>>
>> Well, it needs to understand how some other stuff (especially PGPROC
>> entries) is distributed between nodes. I'm not sure how much of this
>> internal information we want to expose outside core ...
>>
>>> Parts of your code where you assign/define policy could be in one or
>>> more relevant routines of a "numa profile manager", like in an
>>> initProcessRoutine(), and registered in pmroutine struct:
>>>
>>> pmroutine = GetPmRoutineForInitProcess();
>>> if (pmroutine != NULL &&
>>> pmroutine->init_process != NULL)
>>> pmroutine->init_process(MyProc);
>>>
>>> This way it's easier to manage alternative policies, and also to be able
>>> to adjust when hardware and linux kernel changes.
>>>
>>
>> I'm not against making this extensible, in some way. But I still
>> struggle to imagine a reasonable alternative policy, where the external
>> module gets the same information and ends up with a different decision.
>>
>> So what would the alternate policy look like? What use case would the
>> module be supporting?
>
>
> That's the whole point: there are very distinct usages of PostgreSQL in
> the field. And maybe not all of them will require the policy defined by
> PostgreSQL core.
>
> May I ask the reverse: what prevent external modules from taking those
> decisions ? There are already a lot of area where external code can take
> over PostgreSQL processing, like Neon is doing.
>
The complexity of making everything extensible in an arbitrary way. To
make it extensible in a useful, we need to have a reasonably clear idea
what aspects need to be extensible, and what's the goal.
> There are some very early processing for memory setup that I can see as
> a current blocker, and here I'd refer a more compliant NUMA api as
> proposed by Jakub so it's possible to arrange based on workload,
> hardware configuration or other matters. Reworking to get distinct
> segment and all as you do is great, and combo of both approach probably
> of great interest. There is also this weighted interleave discussed and
> probably much more to come in this area in Linux.
>
> I think some points raised already about possible distinct policies, I
> am precisely claiming that it is hard to come with one good policy with
> limited setup options, thus requirement to keep that flexible enough
> (hooks, api, 100 GUc ?).
>
I'm sorry, I don't want to sound too negative, but "I want arbitrary
extensibility" is not a very useful feedback. I've asked you to give
some examples of policies that'd customize some of the NUMA stuff.
> There is an EPYC story here also, given the NUMA setup can vary
> depending on BIOS setup, associated NUMA policy must probably take that
> into account (L3 can be either real cache or 4 extra "local" NUMA nodes
> - with highly distinct access cost from a RAM module).
> Does that change how PostgreSQL will place memory and process? Is it
> important or of interest ?
>
So how exactly would the policy handle this? Right now we're entirely
oblivious to L3, or on-CPU caches in general. We don't even consider the
size of L3 when sizing hash tables in a hashjoin etc.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-10 15:31 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-10 15:31 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Greg Burd <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On 7/9/25 19:23, Andres Freund wrote:
> Hi,
>
> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>> On Jul 9 2025, at 12:35 pm, Andres Freund <[email protected]> wrote:
>>
>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>> entirely. While clocksweep is perhaps minutely slower in a single
>>> thread than
>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>> rare to be bottlenecked on clock sweep speed for a single thread
>>> (rather then
>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>>
>> Hey Andres, thanks for spending time on this. I've worked before on
>> freelist implementations (last one in LMDB) and I think you're onto
>> something. I think it's an innovative idea and that the speed
>> difference will either be lost in the noise or potentially entirely
>> mitigated by avoiding duplicate work.
>
> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
> perform better because it doesn't need to maintain the freelist anymore...
>
>
>>> Also needing to switch between getting buffers from the freelist and
>>> the sweep
>>> makes the code more expensive. I think just having the buffer in the sweep,
>>> with a refcount / usagecount of zero would suffice.
>>
>> If you're not already coding this, I'll jump in. :)
>
> My experimental patch is literally a four character addition ;), namely adding
> "0 &&" to the relevant code in StrategyGetBuffer().
>
> Obviously a real patch would need to do some more work than that. Feel free
> to take on that project, I am not planning on tackling that in near term.
>
>
> There's other things around this that could use some attention. It's not hard
> to see clock sweep be a bottleneck in concurrent workloads - partially due to
> the shared maintenance of the clock hand. A NUMAed clock sweep would address
> that. However, we also maintain StrategyControl->numBufferAllocs, which is a
> significant contention point and would not necessarily be removed by a
> NUMAificiation of the clock sweep.
>
Wouldn't it make sense to partition the numBufferAllocs too, though? I
don't remember if my hacky experimental patch NUMA-partitioning did that
or I just thought about doing that, but why wouldn't that be enough?
Places that need the "total" count would have to sum the counters, but
it seemed to me most of the places would be fine with the "local" count
for that partition. If we also make sure to "sync" the clocksweeps so as
to not work on just a single partition, that might be enough ...
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-11 16:06 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-11 16:06 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Greg Burd <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
On 2025-07-10 17:31:45 +0200, Tomas Vondra wrote:
> On 7/9/25 19:23, Andres Freund wrote:
> > There's other things around this that could use some attention. It's not hard
> > to see clock sweep be a bottleneck in concurrent workloads - partially due to
> > the shared maintenance of the clock hand. A NUMAed clock sweep would address
> > that. However, we also maintain StrategyControl->numBufferAllocs, which is a
> > significant contention point and would not necessarily be removed by a
> > NUMAificiation of the clock sweep.
> >
>
> Wouldn't it make sense to partition the numBufferAllocs too, though? I
> don't remember if my hacky experimental patch NUMA-partitioning did that
> or I just thought about doing that, but why wouldn't that be enough?
It could be solved together with partitioning, yes - that's what I was trying
to reference with the emphasized bit in "would *not necessarily* be removed by
a NUMAificiation of the clock sweep".
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-11 16:14 Andres Freund <[email protected]>
parent: Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Andres Freund @ 2025-07-11 16:14 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-07-10 14:17:21 +0000, Bertrand Drouvot wrote:
> On Wed, Jul 09, 2025 at 03:42:26PM -0400, Andres Freund wrote:
> > I wonder if we should *increase* the size of shared_buffers whenever huge
> > pages are in use and there's padding space due to the huge page
> > boundaries. Pretty pointless to waste that memory if we can instead use if for
> > the buffer pool. Not that big a deal with 2MB huge pages, but with 1GB huge
> > pages...
>
> I think that makes sense, except maybe for operations that need to scan
> the whole buffer pool (i.e related to BUF_DROP_FULL_SCAN_THRESHOLD)?
I don't think the increases here are big enough for that to matter, unless
perhaps you're using 1GB huge pages. But if you're concerned about dropping
tables very fast (i.e. you're running schema change heavy regression tests),
you're not going to use 1GB huge pages.
> > > 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
> > >
> > > Another area that seems like it might benefit from NUMA is PGPROC, so I
> > > gave it a try. It turned out somewhat challenging. Similarly to buffers
> > > we have two pieces that need to be located in a coordinated way - PGPROC
> > > entries and fast-path arrays. But we can't use the same approach as for
> > > buffers/descriptors, because
> > >
> > > (a) Neither of those pieces aligns with memory page size (PGPROC is
> > > ~900B, fast-path arrays are variable length).
> >
> > > (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> > > rather high max_connections before we use multiple huge pages.
> >
> > Right now sizeof(PGPROC) happens to be multiple of 64 (i.e. the most common
> > cache line size)
>
> Oh right, it's currently 832 bytes and the patch extends that to 840 bytes.
I don't think the patch itself is the problem - it really is just happenstance
that it's a multiple of the line size right now. And it's not on common Armv8
platforms...
> With a bit of reordering:
>
> That could be back to 832 (the order does not make sense logically speaking
> though).
I don't think shrinking the size in a one-off way just to keep the
"accidental" size-is-multiple-of-64 property is promising. It'll just get
broken again. I think we should:
a) pad the size of PGPROC to a cache line (or even to a subsequent power of 2,
to make array access cheaper, right now that involves actual
multiplications rather than shifts or indexed `lea` instructions).
That's probably just a pg_attribute_aligned
b) Reorder PGPROC to separate frequently modified from almost-read-only data,
to increase cache hit ratio.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-11 17:34 Burd, Greg <[email protected]>
parent: Burd, Greg <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Burd, Greg @ 2025-07-11 17:34 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
> On Jul 10, 2025, at 8:13 AM, Burd, Greg <[email protected]> wrote:
>
>
>> On Jul 9, 2025, at 1:23 PM, Andres Freund <[email protected]> wrote:
>>
>> Hi,
>>
>> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>>> On Jul 9 2025, at 12:35 pm, Andres Freund <[email protected]> wrote:
>>>
>>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>>> entirely. While clocksweep is perhaps minutely slower in a single
>>>> thread than
>>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>>> rare to be bottlenecked on clock sweep speed for a single thread
>>>> (rather then
>>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>>>
>>> Hey Andres, thanks for spending time on this. I've worked before on
>>> freelist implementations (last one in LMDB) and I think you're onto
>>> something. I think it's an innovative idea and that the speed
>>> difference will either be lost in the noise or potentially entirely
>>> mitigated by avoiding duplicate work.
>>
>> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
>> perform better because it doesn't need to maintain the freelist anymore...
>>
>>
>>>> Also needing to switch between getting buffers from the freelist and
>>>> the sweep
>>>> makes the code more expensive. I think just having the buffer in the sweep,
>>>> with a refcount / usagecount of zero would suffice.
>>>
>>> If you're not already coding this, I'll jump in. :)
>>
>> My experimental patch is literally a four character addition ;), namely adding
>> "0 &&" to the relevant code in StrategyGetBuffer().
>>
>> Obviously a real patch would need to do some more work than that. Feel free
>> to take on that project, I am not planning on tackling that in near term.
>>
>
> I started on this last night, making good progress. Thanks for the inspiration. I'll create a new thread to track the work and cross-reference when I have something reasonable to show (hopefully later today).
>
>> There's other things around this that could use some attention. It's not hard
>> to see clock sweep be a bottleneck in concurrent workloads - partially due to
>> the shared maintenance of the clock hand. A NUMAed clock sweep would address
>> that.
>
> Working on it.
For archival sake, and to tie up loose ends I'll link from here to a new thread I just started that proposes the removal of the freelist and the buffer_strategy_lock [1].
That patch set doesn't address any NUMA-related tasks directly, but it should remove some pain when working in that direction by removing code that requires partitioning and locking and...
best.
-greg
[1] https://postgr.es/m/[email protected]
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-17 21:11 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-17 21:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Greg Burd <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
Here's a v2 of the patch series, with a couple changes:
* I simplified the various freelist partitioning by keeping only the
"node" partitioning (so the cpu/pid strategies are gone). Those were
meant for experimenting, but it made the code more complicated so I
ditched it.
* I changed the freelist partitioning scheme a little bit, based on the
discussion in this thread. Instead of having a single "partition" per
NUMA node, there's not a minimum number of partitions (set to 4). So
even if your system is not NUMA, you'll have 4 of them. If you have 2
nodes, you'll still have 4, and each node will get 2. With 3 nodes we
get 6 partitions (we need 2 per node, and we want to keep the number
equal to keep things simple). Once the number of nodes exceeds 4, the
heuristics switches to one partition per node.
I'm aware there's a discussion about maybe simply removing freelists
entirely. If that happens, this becomes mostly irrelevant, of course.
The code should also make sure the freelists "agree" with how the
earlier patch mapped the buffers to NUMA nodes, i.e. the freelist should
only contain buffers from the "correct" NUMA node, etc. I haven't paid
much attention to this - I believe it should work for "nice" values of
shared buffers (when it evenly divides between nodes). But I'm sure it's
possible to confuse that (won't cause crashes, but inefficiency).
* There's now a patch partitioning clocksweep, using the same scheme as
the freelists. I came to the conclusion it doesn't make much sense to
partition these things differently - I can't think of a reason why that
would be advantageous, and it makes it easier to reason about.
The clocksweep partitioning is somewhat harder, because it affects
BgBufferSync() and related code. With the partitioning we now have
multiple "clock hands" for different ranges of buffers, and the clock
sweep needs to consider that. I modified BgBufferSync to simply loop
through the ClockSweep partitions, and do a small cleanup for each.
It does work, as in "it doesn't crash". But this part definitely needs
review to make sure I got the changes to the "pacing" right.
* This new freelist/clocksweep partitioning scheme is however harder to
disable. I now realize the GUC may quite do the trick, and there even is
not a GUC for the clocksweep. I need to think about this, but I'm not
even how feasible it'd be to have two separate GUCs (because of how
these two pieces are intertwined). For now if you want to test without
the partitioning, you need to skip the patch.
I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
the results are encouraging. For a read-only pgbench (2x shared buffers,
within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
bad considering the patch is more about consistency than raw throughput.
For a read-write pgbench I however saw some strange drops/increases of
throughput. I suspect this might be due to some thinko in the clocksweep
partitioning, but I'll need to take a closer look.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v2-0007-NUMA-pin-backends-to-NUMA-nodes.patch (3.5K, ../../[email protected]/2-v2-0007-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From ca651eb85a6656c79fee5aaabc99e4b772b1b8fe Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v2 7/7] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 9d3e94a7b3a..4c9e55608b2 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -729,6 +729,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ee4684d1b8..3f88659b49f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7b718760248..862341e137e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2156,6 +2156,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index cdeee8dccba..a97741c6707 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.49.0
[text/x-patch] v2-0006-NUMA-interleave-PGPROC-entries.patch (34.8K, ../../[email protected]/3-v2-0006-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From 0d79d2fb6ab9f1d5b0b3f03e500315135329b09e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v2 6/7] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 61 ++--
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 368 +++++++++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 11 +-
11 files changed, 406 insertions(+), 62 deletions(-)
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..087279a6a8e 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 1827e052da7..2ce158ca9bd 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -446,7 +446,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 2418967def6..82158eeb5d6 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1622,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1739,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2223,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2297,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2498,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2724,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2755,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -3006,7 +3005,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3047,7 +3046,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3175,7 +3174,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3218,7 +3217,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3287,7 +3286,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3389,7 +3388,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3454,7 +3453,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3509,7 +3508,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3555,7 +3554,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3584,7 +3583,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3615,7 +3614,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3656,7 +3655,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3719,7 +3718,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3785,7 +3784,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 62f3471448e..c84a2a5f1bc 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..9d3e94a7b3a 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -89,6 +97,12 @@ static void ProcKill(int code, Datum arg);
static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* NUMA */
+static Size get_memory_page_size(void); /* XXX duplicate */
+static void move_to_node(char *startptr, char *endptr,
+ Size mem_page_size, int node);
+static int numa_nodes = -1;
+
/*
* Report shared-memory space needed by PGPROC.
@@ -100,11 +114,40 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * With NUMA, we allocate the PGPROC array in several chunks. With shared
+ * buffers we simply manually assign parts of the buffer array to
+ * different NUMA nodes, and that does the trick. But we can't do that for
+ * PGPROC, as the number of PGPROC entries is much lower, especially with
+ * huge pages. We can fit ~2k entries on a 2MB page, and NUMA does stuff
+ * with page granularity, and the large NUMA systems are likely to use
+ * huge pages. So with sensible max_connections we would not use more than
+ * a single page, which means it gets to a single NUMA node.
+ *
+ * So we allocate PGPROC not as a single array, but one array per NUMA
+ * node, and then one array for aux processes (without NUMA node
+ * assigned). Each array may need up to memory-page-worth of padding,
+ * worst case. So we just add that - it's a bit wasteful, but good enough
+ * for PoC.
+ *
+ * FIXME Should be conditional, but that was causing problems in bootstrap
+ * mode. Or maybe it was because the code that allocates stuff later does
+ * not do that conditionally. Anyway, needs to be fixed.
+ */
+ /* if (numa_procs_interleave) */
+ {
+ int num_nodes = numa_num_configured_nodes();
+ Size mem_page_size = get_memory_page_size();
+
+ size = add_size(size, mul_size((num_nodes + 1), mem_page_size));
+ }
+
return size;
}
@@ -129,6 +172,26 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * Same NUMA-padding logic as in PGProcShmemSize, adding a memory page per
+ * NUMA node - but this way we add two pages per node - one for PGPROC,
+ * one for fast-path arrays. In theory we could make this work just one
+ * page per node, by adding fast-path arrays right after PGPROC entries on
+ * each node. But now we allocate fast-path locks separately - good enough
+ * for PoC.
+ *
+ * FIXME Should be conditional, but that was causing problems in bootstrap
+ * mode. Or maybe it was because the code that allocates stuff later does
+ * not do that conditionally. Anyway, needs to be fixed.
+ */
+ /* if (numa_procs_interleave) */
+ {
+ int num_nodes = numa_num_configured_nodes();
+ Size mem_page_size = get_memory_page_size();
+
+ size = add_size(size, mul_size((num_nodes + 1), mem_page_size));
+ }
+
return size;
}
@@ -191,11 +254,13 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+ int procs_total;
+ int procs_per_node;
/* Used for setup of per-backend fast-path slots. */
char *fpPtr,
@@ -205,6 +270,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -224,6 +291,9 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* one chunk per NUMA node (without NUMA assume 1 node) */
+ numa_nodes = numa_num_configured_nodes();
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,19 +311,108 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it now.
+ */
+ procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+ procs_total = 0;
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ PGPROC *procs_node;
+
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int count_node = Min(procs_per_node, MaxBackends - procs_total);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(mem_page_size, ptr);
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+ ptr = (char *) ptr + count_node * sizeof(PGPROC);
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (j = 0; j < count_node; j++)
+ {
+ procs_node[j].numa_node = i;
+ procs_node[j].procnumber = procs_total;
+
+ ProcGlobal->allProcs[procs_total++] = &procs_node[j];
+ }
+
+ move_to_node((char *) procs_node, ptr, mem_page_size, i);
+ }
+
+ /*
+ * also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * don't assign those to any NUMA node)
+ *
+ * XXX Mostly duplicate of preceding block, could be reused.
+ */
+ {
+ PGPROC *procs_node;
+ int count_node = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /*
+ * Make sure to align PGPROC array to memory page (it may not be
+ * aligned). We won't assign this to any NUMA node, but we still don't
+ * want it to interfere with the preceding chunk (for the last NUMA
+ * node).
+ */
+ ptr = (char *) TYPEALIGN(mem_page_size, ptr);
+
+ procs_node = (PGPROC *) ptr;
+ ptr = (char *) ptr + count_node * sizeof(PGPROC);
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ /* now add the PGPROC pointers to allProcs */
+ for (j = 0; j < count_node; j++)
+ {
+ procs_node[j].numa_node = -1;
+ procs_node[j].procnumber = procs_total;
+
+ ProcGlobal->allProcs[procs_total++] = &procs_node[j];
+ }
+ }
+
+ /* we should have allocated the expected number of PGPROC entries */
+ Assert(procs_total == TotalProcs);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,23 +445,100 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /* reset the count */
+ procs_total = 0;
+
+ /*
+ * Mimic the same logic as above, but for fast-path locking.
+ */
+ for (i = 0; i < numa_nodes; i++)
{
- PGPROC *proc = &procs[i];
+ char *startptr;
+ char *endptr;
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int procs_node = Min(procs_per_node, MaxBackends - procs_total);
+
+ /* align to memory page, to make move_pages possible */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
+
+ startptr = fpPtr;
+ endptr = fpPtr + procs_node * (fpLockBitsSize + fpRelIdSize);
+
+ move_to_node(startptr, endptr, mem_page_size, i);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Now point the PGPROC entries to the fast-path arrays, and also
+ * advance the fpPtr.
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ for (j = 0; j < procs_node; j++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[procs_total++];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == i);
+ Assert(proc->procnumber == (procs_total - 1));
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We
+ * interleave the two arrays, to (hopefully) get some locality for
+ * each backend.
+ */
+ proc->fpLockBits = (uint64 *) fpPtr;
+ fpPtr += fpLockBitsSize;
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ proc->fpRelId = (Oid *) fpPtr;
+ fpPtr += fpRelIdSize;
- Assert(fpPtr <= fpEndPtr);
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(fpPtr == endptr);
+ }
+
+ /* auxiliary processes / prepared xacts */
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ int procs_node = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* align to memory page, to make move_pages possible */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
+
+ /* now point the PGPROC entries to the fast-path arrays */
+ for (j = 0; j < procs_node; j++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[procs_total++];
+
+ /* cross-check we got PGPROC with no NUMA node assigned */
+ Assert(proc->numa_node == -1);
+ Assert(proc->procnumber == (procs_total - 1));
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We
+ * interleave the two arrays, to (hopefully) get some locality for
+ * each backend.
+ */
+ proc->fpLockBits = (uint64 *) fpPtr;
+ fpPtr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) fpPtr;
+ fpPtr += fpRelIdSize;
+
+ Assert(fpPtr <= fpEndPtr);
+ }
+ }
+
+ /* Should have consumed exactly the expected amount of fast-path memory. */
+ Assert(fpPtr <= fpEndPtr);
+
+ /* make sure we allocated the expected number of PGPROC entries */
+ Assert(procs_total == TotalProcs);
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -366,15 +602,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +668,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2259,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2334,60 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * move_to_node
+ * move all pages in the given range to the requested NUMA node
+ *
+ * XXX This is expected to only process fairly small number of pages, so no
+ * need to do batching etc. Just move pages one by one.
+ */
+static void
+move_to_node(char *startptr, char *endptr, Size mem_page_size, int node)
+{
+ while (startptr < endptr)
+ {
+ int r,
+ status;
+
+ r = numa_move_pages(0, 1, (void **) &startptr, &node, &status, 0);
+
+ if (r != 0)
+ elog(WARNING, "failed to move page to NUMA node %d (r = %d, status = %d)",
+ node, r, status);
+
+ startptr += mem_page_size;
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a11bc71a386..6ee4684d1b8 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0552ed62cc7..7b718760248 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2146,6 +2146,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 66baf2bf33e..cdeee8dccba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 9f9b3fcfbf1..5cb1632718e 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -194,6 +194,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -319,6 +321,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -383,7 +388,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -435,8 +440,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
--
2.49.0
[text/x-patch] v2-0005-NUMA-clockweep-partitioning.patch (33.2K, ../../[email protected]/4-v2-0005-NUMA-clockweep-partitioning.patch)
download | inline diff:
From c4d51ab87b92f9900e37d42cf74980e87b648a56 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v2 5/7] NUMA: clockweep partitioning
---
src/backend/storage/buffer/bufmgr.c | 473 ++++++++++++++------------
src/backend/storage/buffer/freelist.c | 202 ++++++++---
src/include/storage/buf_internals.h | 4 +-
3 files changed, 424 insertions(+), 255 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5922689fe5d..3d6c834d77c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3587,6 +3587,23 @@ BufferSync(int flags)
TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_scan);
}
+/*
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
+ *
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
+ *
+ * XXX might be better to have a per-partition struct with all the info
+ */
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
/*
* BgBufferSync -- Write out some dirty buffers in the pool.
*
@@ -3602,55 +3619,24 @@ bool
BgBufferSync(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
- int strategy_buf_id;
- uint32 strategy_passes;
uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
-
- /* Moving averages of allocation rate and clean-buffer density */
- static float smoothed_alloc = 0;
- static float smoothed_density = 10.0;
-
- /* Potentially these could be tunables, but for now, not */
- float smoothing_samples = 16;
- float scan_whole_pool_milliseconds = 120000.0;
-
- /* Used to compute how far we scan ahead */
- long strategy_delta;
- int bufs_to_lap;
- int bufs_ahead;
- float scans_per_alloc;
- int reusable_buffers_est;
- int upcoming_alloc_est;
- int min_scan_buffers;
-
- /* Variables for the scanning loop proper */
- int num_to_scan;
- int num_written;
- int reusable_buffers;
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
- /* Variables for final smoothed_density update */
- long new_strategy_delta;
- uint32 new_recent_alloc;
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
- /*
- * Find out where the freelist clock sweep currently is, and how many
- * buffer allocations have happened since our last call.
- */
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
/* Report buffer alloc counts to pgstat */
PendingBgWriterStats.buf_alloc += recent_alloc;
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -3663,223 +3649,282 @@ BgBufferSync(WritebackContext *wb_context)
}
/*
- * Compute strategy_delta = how many buffers have been scanned by the
- * clock sweep since last time. If first time through, assume none. Then
- * see if we are still ahead of the clock sweep, and if so, how many
- * buffers we could scan before we'd catch up with it and "lap" it. Note:
- * weird-looking coding of xxx_passes comparisons are to avoid bogus
- * behavior when the passes counts wrap around.
- */
- if (saved_info_valid)
- {
- int32 passes_delta = strategy_passes - prev_strategy_passes;
-
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that
+ * we don't start from partition 0 all the time? Perhaps not entirely,
+ * but at least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* info obtained from freelist.c */
+ int strategy_buf_id;
+ uint32 strategy_passes;
+
+ /* Moving averages of allocation rate and clean-buffer density */
+ static float smoothed_alloc = 0;
+ static float smoothed_density = 10.0;
+
+ /* Potentially these could be tunables, but for now, not */
+ float smoothing_samples = 16;
+ float scan_whole_pool_milliseconds = 120000.0;
+
+ /* Used to compute how far we scan ahead */
+ long strategy_delta;
+ int bufs_to_lap;
+ int bufs_ahead;
+ float scans_per_alloc;
+ int reusable_buffers_est;
+ int upcoming_alloc_est;
+ int min_scan_buffers;
+
+ /* Variables for the scanning loop proper */
+ int num_to_scan;
+ int num_written;
+ int reusable_buffers;
+
+ /* Variables for final smoothed_density update */
+ long new_strategy_delta;
+ uint32 new_recent_alloc;
+
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
- Assert(strategy_delta >= 0);
+ /*
+ * Find out where the freelist clock sweep currently is, and how many
+ * buffer allocations have happened since our last call.
+ */
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
- if ((int32) (next_passes - strategy_passes) > 0)
+ /*
+ * Compute strategy_delta = how many buffers have been scanned by the
+ * clock sweep since last time. If first time through, assume none. Then
+ * see if we are still ahead of the clock sweep, and if so, how many
+ * buffers we could scan before we'd catch up with it and "lap" it. Note:
+ * weird-looking coding of xxx_passes comparisons are to avoid bogus
+ * behavior when the passes counts wrap around.
+ */
+ if (saved_info_valid)
{
- /* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
+
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
+
+ Assert(strategy_delta >= 0);
+
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
+ {
+ /* we're one pass ahead of the strategy point */
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
#endif
- }
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
- {
- /* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ }
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
+ {
+ /* on same pass, but ahead or at least not behind */
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
+#ifdef BGW_DEBUG
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
+#endif
+ }
+ else
+ {
+ /*
+ * We're behind, so skip forward to the strategy point and start
+ * cleaning from there.
+ */
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta);
#endif
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
+ }
}
else
{
/*
- * We're behind, so skip forward to the strategy point and start
- * cleaning from there.
+ * Initializing at startup or after LRU scanning had been off. Always
+ * start at the strategy point.
*/
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta);
+ elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
+ strategy_passes, strategy_buf_id);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ strategy_delta = 0;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
- }
- else
- {
- /*
- * Initializing at startup or after LRU scanning had been off. Always
- * start at the strategy point.
- */
-#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
- strategy_passes, strategy_buf_id);
-#endif
- strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
- }
- /* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ /* Update saved info for next time */
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ // FIXME has to happen after all partitions
+ // saved_info_valid = true;
- /*
- * Compute how many buffers had to be scanned for each new allocation, ie,
- * 1/density of reusable buffers, and track a moving average of that.
- *
- * If the strategy point didn't move, we don't update the density estimate
- */
- if (strategy_delta > 0 && recent_alloc > 0)
- {
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
- }
+ /*
+ * Compute how many buffers had to be scanned for each new allocation, ie,
+ * 1/density of reusable buffers, and track a moving average of that.
+ *
+ * If the strategy point didn't move, we don't update the density estimate
+ */
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
+ {
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
+ }
- /*
- * Estimate how many reusable buffers there are between the current
- * strategy point and where we've scanned ahead to, based on the smoothed
- * density estimate.
- */
- bufs_ahead = NBuffers - bufs_to_lap;
- reusable_buffers_est = (float) bufs_ahead / smoothed_density;
+ /*
+ * Estimate how many reusable buffers there are between the current
+ * strategy point and where we've scanned ahead to, based on the smoothed
+ * density estimate.
+ */
+ bufs_ahead = num_buffers - bufs_to_lap;
+ reusable_buffers_est = (float) bufs_ahead / smoothed_density;
- /*
- * Track a moving average of recent buffer allocations. Here, rather than
- * a true average we want a fast-attack, slow-decline behavior: we
- * immediately follow any increase.
- */
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
- else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
- smoothing_samples;
+ /*
+ * Track a moving average of recent buffer allocations. Here, rather than
+ * a true average we want a fast-attack, slow-decline behavior: we
+ * immediately follow any increase.
+ */
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
+ else
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
+ smoothing_samples;
- /* Scale the estimate by a GUC to allow more aggressive tuning. */
- upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
+ /* Scale the estimate by a GUC to allow more aggressive tuning. */
+ upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
- /*
- * If recent_alloc remains at zero for many cycles, smoothed_alloc will
- * eventually underflow to zero, and the underflows produce annoying
- * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
- * zero, there's no point in tracking smaller and smaller values of
- * smoothed_alloc, so just reset it to exactly zero to avoid this
- * syndrome. It will pop back up as soon as recent_alloc increases.
- */
- if (upcoming_alloc_est == 0)
- smoothed_alloc = 0;
+ /*
+ * If recent_alloc remains at zero for many cycles, smoothed_alloc will
+ * eventually underflow to zero, and the underflows produce annoying
+ * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
+ * zero, there's no point in tracking smaller and smaller values of
+ * smoothed_alloc, so just reset it to exactly zero to avoid this
+ * syndrome. It will pop back up as soon as recent_alloc increases.
+ */
+ if (upcoming_alloc_est == 0)
+ smoothed_alloc = 0;
- /*
- * Even in cases where there's been little or no buffer allocation
- * activity, we want to make a small amount of progress through the buffer
- * cache so that as many reusable buffers as possible are clean after an
- * idle period.
- *
- * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
- * the BGW will be called during the scan_whole_pool time; slice the
- * buffer pool into that many sections.
- */
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ /*
+ * Even in cases where there's been little or no buffer allocation
+ * activity, we want to make a small amount of progress through the buffer
+ * cache so that as many reusable buffers as possible are clean after an
+ * idle period.
+ *
+ * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
+ * the BGW will be called during the scan_whole_pool time; slice the
+ * buffer pool into that many sections.
+ */
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
- if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
- {
+ if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
+ {
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
- upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
+ elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
+ upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
#endif
- upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
- }
-
- /*
- * Now write out dirty reusable buffers, working forward from the
- * next_to_clean point, until we have lapped the strategy scan, or cleaned
- * enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.
- */
+ upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
+ }
- num_to_scan = bufs_to_lap;
- num_written = 0;
- reusable_buffers = reusable_buffers_est;
+ /*
+ * Now write out dirty reusable buffers, working forward from the
+ * next_to_clean point, until we have lapped the strategy scan, or cleaned
+ * enough buffers to match our estimate of the next cycle's allocation
+ * requirements, or hit the bgwriter_lru_maxpages limit.
+ */
- /* Execute the LRU scan */
- while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
- {
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ num_to_scan = bufs_to_lap;
+ num_written = 0;
+ reusable_buffers = reusable_buffers_est;
- if (++next_to_clean >= NBuffers)
+ /* Execute the LRU scan */
+ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- next_to_clean = 0;
- next_passes++;
- }
- num_to_scan--;
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
+ wb_context);
- if (sync_state & BUF_WRITTEN)
- {
- reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- PendingBgWriterStats.maxwritten_clean++;
- break;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
+ }
+ num_to_scan--;
+
+ if (sync_state & BUF_WRITTEN)
+ {
+ reusable_buffers++;
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
+ {
+ PendingBgWriterStats.maxwritten_clean++;
+ break;
+ }
}
+ else if (sync_state & BUF_REUSABLE)
+ reusable_buffers++;
}
- else if (sync_state & BUF_REUSABLE)
- reusable_buffers++;
- }
- PendingBgWriterStats.buf_written_clean += num_written;
+ PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
- elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
- smoothed_density, reusable_buffers_est, upcoming_alloc_est,
- bufs_to_lap - num_to_scan,
- num_written,
- reusable_buffers - reusable_buffers_est);
+ elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
+ smoothed_density, reusable_buffers_est, upcoming_alloc_est,
+ bufs_to_lap - num_to_scan,
+ num_written,
+ reusable_buffers - reusable_buffers_est);
#endif
- /*
- * Consider the above scan as being like a new allocation scan.
- * Characterize its density and update the smoothed one based on it. This
- * effectively halves the moving average period in cases where both the
- * strategy and the background writer are doing some useful scanning,
- * which is helpful because a long memory isn't as desirable on the
- * density estimates.
- */
- new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
- {
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
+ /*
+ * Consider the above scan as being like a new allocation scan.
+ * Characterize its density and update the smoothed one based on it. This
+ * effectively halves the moving average period in cases where both the
+ * strategy and the background writer are doing some useful scanning,
+ * which is helpful because a long memory isn't as desirable on the
+ * density estimates.
+ */
+ new_strategy_delta = bufs_to_lap - num_to_scan;
+ new_recent_alloc = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ {
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
- scans_per_alloc, smoothed_density);
+ elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
+ new_recent_alloc, new_strategy_delta,
+ scans_per_alloc, smoothed_density);
#endif
+ }
+
+ /* hibernate if all partitions can hibernate */
+ hibernate &= (bufs_to_lap == 0 && recent_alloc_partition == 0);
}
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e38e5c7ec3d..1827e052da7 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -63,17 +63,27 @@ typedef struct BufferStrategyFreelist
#define MIN_FREELIST_PARTITIONS 4
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* Clock sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -83,6 +93,15 @@ typedef struct
*/
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
/*
* Bgworker process to be notified upon activity or -1 if none. See
@@ -99,6 +118,9 @@ typedef struct
int num_partitions_groups; /* effectively num of NUMA nodes */
int num_partitions_per_group;
+ /* clocksweep partitions */
+ ClockSweep *sweeps;
+
BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
@@ -152,6 +174,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -163,6 +186,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -170,14 +194,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -203,19 +227,23 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
}
/*
@@ -289,6 +317,28 @@ calculate_partition_index()
return index;
}
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
+}
+
/*
* ChooseFreeList
* Pick the buffer freelist to use, depending on the CPU and NUMA node.
@@ -404,7 +454,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -475,13 +525,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Nothing on the freelist, so run the "clock sweep" algorithm
*
- * XXX Should we also make this NUMA-aware, to only access buffers from
- * the same NUMA node? That'd probably mean we need to make the clock
- * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
- * subset of buffers. But that also means each process could "sweep" only
- * a fraction of buffers, even if the other buffers are better candidates
- * for eviction. Would that also mean we'd have multiple bgwriters, one
- * for each node, or would one bgwriter handle all of that?
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That
+ * means it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists
+ * or other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
*/
trycounter = NBuffers;
for (;;)
@@ -563,6 +617,41 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncStart -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ *num_buf_alloc += pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -570,37 +659,44 @@ StrategyFreeBuffer(BufferDesc *buf)
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -696,6 +792,10 @@ StrategyShmemSize(void)
size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
num_partitions)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -714,6 +814,7 @@ StrategyInitialize(bool init)
int num_partitions;
int num_partitions_per_group;
+ char *ptr;
/* */
num_partitions = calculate_partition_count(strategy_nnodes);
@@ -736,7 +837,8 @@ StrategyInitialize(bool init)
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
- MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -758,12 +860,32 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* have to point the sweeps array to right after the freelists */
+ ptr = (char *) StrategyControl +
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions);
+ StrategyControl->sweeps = (ClockSweep *) ptr;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not
+ * a perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = numBuffers;
+ StrategyControl->sweeps[i].firstBuffer = (numBuffers * i);
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 52a71b138f7..b50f9458156 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -448,7 +448,9 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
--
2.49.0
[text/x-patch] v2-0004-NUMA-partition-buffer-freelist.patch (19.8K, ../../[email protected]/5-v2-0004-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From d67278a64983b5f2eb5e408a51e9516aa8fd2264 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v2 4/7] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
There are four strategies, specified by GUC numa_partition_freelist
* none - single long freelist, should work just like now
* node - one freelist per NUMA node, with only buffers from that node
* cpu - one freelist per CPU
* pid - freelist determined by PID (same number of freelists as 'cpu')
When allocating a buffer, it's taken from the correct freelist (e.g.
same NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
src/backend/storage/buffer/buf_init.c | 4 +-
src/backend/storage/buffer/freelist.c | 372 ++++++++++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 8 +
6 files changed, 367 insertions(+), 29 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 2ad34624c49..920f1a32a8f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -543,8 +543,8 @@ pg_numa_interleave_memory(char *startptr, char *endptr,
* XXX no return value, to make this fail on error, has to use
* numa_set_strict
*
- * XXX Should we still touch the memory first, like with numa_move_pages,
- * or is that not necessary?
+ * XXX Should we still touch the memory first, like with
+ * numa_move_pages, or is that not necessary?
*/
numa_tonode_memory(ptr, sz, node);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..e38e5c7ec3d 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,52 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
+
+/*
+ * The minimum number of partitions we want to have. We want at least this
+ * number of partitions, even on non-NUMA system, as it helps with contention
+ * for buffers. But with multiple NUMA nodes, we want a separate partition per
+ * node. But we may get multiple partitions per node, for low node count.
+ *
+ * With multiple partitions per NUMA node, we pick the partition based on CPU
+ * (or some other parameter).
+ */
+#define MIN_FREELIST_PARTITIONS 4
/*
* The shared freelist control information.
@@ -39,8 +77,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,13 +87,38 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+ int num_partitions_groups; /* effectively num of NUMA nodes */
+ int num_partitions_per_group;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
static BufferStrategyControl *StrategyControl = NULL;
+/*
+ * XXX shouldn't this be in BufferStrategyControl? Probably not, we need to
+ * calculate it during sizing, and perhaps it could change before the memory
+ * gets allocated (so we need to remember the values).
+ *
+ * XXX We should probably have a fixed number of partitions, and map the
+ * NUMA nodes to them, somehow (i.e. each node would get some subset of
+ * partitions). Similar to NUM_LOCK_PARTITIONS.
+ *
+ * XXX We don't use the ncpus, really.
+ */
+static int strategy_ncpus;
+static int strategy_nnodes;
+
/*
* Private (non-shared) state for managing a ring of shared buffers to re-use.
* This is currently the only kind of BufferAccessStrategy object, but someday
@@ -157,6 +218,104 @@ ClockSweepTick(void)
return victim;
}
+/*
+ * Size the clocksweep partitions. At least one partition per NUMA node,
+ * but at least MIN_FREELIST_PARTITIONS partitions in total.
+*/
+static int
+calculate_partition_count(int num_nodes)
+{
+ int num_per_node = 1;
+
+ while (num_per_node * num_nodes < MIN_FREELIST_PARTITIONS)
+ num_per_node++;
+
+ return (num_nodes * num_per_node);
+}
+
+static int
+calculate_partition_index()
+{
+ int rc;
+ unsigned cpu;
+ unsigned node;
+ int index;
+
+ Assert(StrategyControl->num_partitions_groups == strategy_nnodes);
+
+ Assert(StrategyControl->num_partitions ==
+ (strategy_nnodes * StrategyControl->num_partitions_per_group));
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building
+ * the partitions. Maybe if we allow this (e.g. due to support adjusting
+ * the NUMA stuff at runtime), we should just do our best to minimize
+ * the conflicts somehow. But it'll make the mapping harder, so for now
+ * we ignore it.
+ */
+ if (node > strategy_nnodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, strategy_nnodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_group == 1)
+ {
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_partitions_groups);
+ index_part = (cpu % StrategyControl->num_partitions_per_group);
+
+ index = (index_group * StrategyControl->num_partitions_per_group)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ int index = calculate_partition_index();
+ return &StrategyControl->freelists[index];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +327,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < strategy_nnodes; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +355,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +422,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +472,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -356,7 +533,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ /*
+ * We don't want to call ChooseFreeList() again, because we might get a
+ * completely different freelist - either a different partition in the
+ * same group, or even a different group if the NUMA node changed. But
+ * we can calculate the proper freelist from the buffer id.
+ */
+ int index = (BufferGetNode(buf->buf_id) * StrategyControl->num_partitions_per_group)
+ + (buf->buf_id % StrategyControl->num_partitions_per_group);
+
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[index];
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +556,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +624,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int node = 0; node < strategy_nnodes; node++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[node];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "freelist %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ node,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -445,12 +673,28 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ /* FIXME */
+#ifdef USE_LIBNUMA
+ strategy_ncpus = numa_num_task_cpus();
+ strategy_nnodes = numa_num_task_nodes();
+#else
+ strategy_ncpus = 1;
+ strategy_nnodes = 1;
+#endif
+
+ num_partitions = calculate_partition_count(strategy_nnodes);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /* size of freelist partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ num_partitions)));
return size;
}
@@ -466,6 +710,13 @@ void
StrategyInitialize(bool init)
{
bool found;
+ int buffers_per_partition;
+
+ int num_partitions;
+ int num_partitions_per_group;
+
+ /* */
+ num_partitions = calculate_partition_count(strategy_nnodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -484,23 +735,28 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
&found);
if (!found)
{
+ int32 numBuffers = NBuffers / num_partitions;
+
+ while (numBuffers * num_partitions < NBuffers)
+ numBuffers++;
+
+ Assert(numBuffers * num_partitions == NBuffers);
+
/*
* Only done once, usually in postmaster
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +767,68 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % strategy_nnodes == 0);
+
+ num_partitions_per_group = (num_partitions / strategy_nnodes);
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_partitions_groups = strategy_nnodes;
+ StrategyControl->num_partitions_per_group = num_partitions_per_group;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < num_partitions; nfreelist++)
+ {
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+ }
+
+ /* buffers per partition */
+ buffers_per_partition = (NBuffers / num_partitions);
+
+ elog(LOG, "NBuffers: %d, nodes %d, ncpus: %d, divide: %d, remain: %d",
+ NBuffers, strategy_nnodes, strategy_ncpus,
+ buffers_per_partition, NBuffers - (num_partitions * buffers_per_partition));
+
+ /*
+ * Walk through the buffers, add them to the correct list. Walk from
+ * the end, because we're adding the buffers to the beginning.
+ */
+ for (int i = NBuffers - 1; i >= 0; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+ BufferStrategyFreelist *freelist;
+ int node;
+ int index;
+
+ /*
+ * Split the freelist into partitions, if needed (or just keep the
+ * freelist we already built in BufferManagerShmemInit().
+ */
+
+ /* determine NUMA node for buffer, this determines the group */
+ node = BufferGetNode(i);
+
+ /* now calculate the actual freelist index */
+ index = node * num_partitions_per_group + (i % num_partitions_per_group);
+
+ /* add to the right freelist */
+ freelist = &StrategyControl->freelists[index];
+
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
}
else
Assert(!init);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..a11bc71a386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+bool numa_partition_freelist = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a21f20800fb..0552ed62cc7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2136,6 +2136,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..66baf2bf33e 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index c257c8a1c20..efb7e28c10f 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -93,6 +93,14 @@ typedef enum ExtendBufferedFlags
EB_LOCK_TARGET = (1 << 5),
} ExtendBufferedFlags;
+typedef enum FreelistPartitionMode
+{
+ FREELIST_PARTITION_NONE,
+ FREELIST_PARTITION_NODE,
+ FREELIST_PARTITION_CPU,
+ FREELIST_PARTITION_PID,
+} FreelistPartitionMode;
+
/*
* Some functions identify relations either by relation or smgr +
* relpersistence. Used via the BMR_REL()/BMR_SMGR() macros below. This
--
2.49.0
[text/x-patch] v2-0003-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../../[email protected]/6-v2-0003-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From 2faefc2d10dcd9e31e96be5565e82d1904bd7280 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v2 3/7] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.49.0
[text/x-patch] v2-0002-NUMA-localalloc.patch (3.7K, ../../[email protected]/7-v2-0002-NUMA-localalloc.patch)
download | inline diff:
From c0acd3385fa961e56eb435b85bb021e7ce9e2cb8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v2 2/7] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 16 ++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 28 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 43b4dbccc3d..d11936691b2 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,18 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9570087aa60..a21f20800fb 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.49.0
[text/x-patch] v2-0001-NUMA-interleaving-buffers.patch (26.9K, ../../[email protected]/8-v2-0001-NUMA-interleaving-buffers.patch)
download | inline diff:
From 1eab6285dab1fdc78d80f6054ec3278624a662f1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 6 May 2025 21:12:21 +0200
Subject: [PATCH v2 1/7] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure not to
split a buffer on different NUMA nodes (which with the regular
interleaving is guaranteed to happen, unless when using huge pages). The
patch performs "explicit" interleaving, so that buffers are not split
like this.
The patch maps both buffers and buffer descriptors, so that the buffer
and it's buffer descriptor end up on the same NUMA node.
The mapping happens in larger chunks (see choose_chunk_items). This is
required to handle buffer descriptors (which are smaller than buffers),
and it should also help to reduce the number of mappings. Most NUMA
systems will use 1GB chunks, unless using very small shared buffers.
Notes:
* The feature is enabled by numa_buffers_interleave GUC (false by default)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
src/backend/storage/buffer/buf_init.c | 384 +++++++++++++++++++++++++-
src/backend/storage/buffer/bufmgr.c | 1 +
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/bin/pgbench/pgbench.c | 67 ++---
src/include/miscadmin.h | 2 +
src/include/storage/bufmgr.h | 1 +
7 files changed, 427 insertions(+), 41 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..2ad34624c49 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -25,6 +33,19 @@ WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+static Size get_memory_page_size(void);
+static int64 choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes);
+static void pg_numa_interleave_memory(char *startptr, char *endptr,
+ Size mem_page_size, Size chunk_size,
+ int num_nodes);
+
+/* number of buffers allocated on the same NUMA node */
+static int64 numa_chunk_buffers = -1;
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1;
+
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -71,18 +92,80 @@ BufferManagerShmemInit(void)
foundDescs,
foundIOCV,
foundBufCkpt;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +195,63 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available
+ * NUMA nodes. We can't use the regular interleaving, because with
+ * regular memory pages (smaller than BLCKSZ) we'd split all buffers
+ * to multiple NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have
+ * mapping for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the
+ * same NUMA node, as calculated by choose_chunk_buffers(). We try to
+ * keep the chunks large enough to work both for buffers and buffer
+ * descriptors, but not too large. See the comments at
+ * choose_chunk_buffers() for details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ */
+ if (numa_buffers_interleave)
+ {
+ char *startptr,
+ *endptr;
+ Size chunk_size;
+
+ numa_nodes = numa_num_configured_nodes();
+
+ numa_chunk_buffers
+ = choose_chunk_buffers(NBuffers, mem_page_size, numa_nodes);
+
+ elog(LOG, "BufferManagerShmemInit num_nodes %d chunk_buffers %ld",
+ numa_nodes, numa_chunk_buffers);
+
+ /* first map buffers */
+ startptr = BufferBlocks;
+ endptr = startptr + ((Size) NBuffers) * BLCKSZ;
+ chunk_size = (numa_chunk_buffers * BLCKSZ);
+
+ pg_numa_interleave_memory(startptr, endptr,
+ mem_page_size,
+ chunk_size,
+ numa_nodes);
+
+ /* now do the same for buffer descriptors */
+ startptr = (char *) BufferDescriptors;
+ endptr = startptr + ((Size) NBuffers) * sizeof(BufferDescPadded);
+ chunk_size = (numa_chunk_buffers * sizeof(BufferDescPadded));
+
+ pg_numa_interleave_memory(startptr, endptr,
+ mem_page_size,
+ chunk_size,
+ numa_nodes);
+ }
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +284,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +297,72 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ Size mem_page_size;
+
+ /* XXX why does IsUnderPostmaster matter? */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(mem_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(mem_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -186,3 +379,178 @@ BufferManagerShmemSize(void)
return size;
}
+
+/*
+ * choose_chunk_buffers
+ * choose the number of buffers allocated to a NUMA node at once
+ *
+ * We don't map shared buffers to NUMA nodes one by one, but in larger chunks.
+ * This is both for efficiency reasons (fewer mappings), and also because we
+ * want to map buffer descriptors too - and descriptors are much smaller. So
+ * we pick a number that's high enough for descriptors to use whole pages.
+ *
+ * We also want to keep buffers somehow evenly distributed on nodes, with
+ * about NBuffers/nodes per node. So we don't use chunks larger than this,
+ * to keep it as fair as possible (the chunk size is a possible difference
+ * between memory allocated to different NUMA nodes).
+ *
+ * It's possible shared buffers are so small this is not possible (i.e.
+ * it's less than chunk_size). But sensible NUMA systems will use a lot
+ * of memory, so this is unlikely.
+ *
+ * We simply print a warning about the misbalance, and that's it.
+ *
+ * XXX It'd be good to ensure the chunk size is a power-of-2, because then
+ * we could calculate the NUMA node simply by shift/modulo, while now we
+ * have to do a division. But we don't know how many buffers and buffer
+ * descriptors fits into a memory page. It may not be a power-of-2.
+ */
+static int64
+choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes)
+{
+ int64 num_items;
+ int64 max_items;
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(mem_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % mem_page_size) == 0) || ((mem_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of items to fill a memory page with descriptors and
+ * blocks. The NUMA allocates memory in pages, and we need to do that for
+ * both buffers and descriptors.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ * But it's clearer this way.
+ */
+ num_items = Max(mem_page_size / sizeof(BufferDescPadded),
+ mem_page_size / BLCKSZ);
+
+ /*
+ * We shouldn't use chunks larger than NBuffers/num_nodes, because with
+ * larger chunks the last NUMA node would end up with much less memory (or
+ * no memory at all).
+ */
+ max_items = (NBuffers / num_nodes);
+
+ /*
+ * Did we already exceed the maximum desirable chunk size? That is, will
+ * the last node get less than one whole chunk (or no memory at all)?
+ */
+ if (num_items > max_items)
+ elog(WARNING, "choose_chunk_buffers: chunk items exceeds max (%ld > %ld)",
+ num_items, max_items);
+
+ /* grow the chunk size until we hit the max limit. */
+ while (2 * num_items <= max_items)
+ num_items *= 2;
+
+ /*
+ * XXX It's not difficult to construct cases where we end up with not
+ * quite balanced distribution. For example, with shared_buffers=10GB and
+ * 4 NUMA nodes, we end up with 2GB chunks, which means the first node
+ * gets 4GB, and the three other nodes get 2GB each.
+ *
+ * We could be smarter, and try to get more balanced distribution. We
+ * could simply reduce max_items e.g. to
+ *
+ * max_items = (NBuffers / num_nodes) / 4;
+ *
+ * in which cases we'd end up with 512MB chunks, and each nodes would get
+ * the same 2.5GB chunk. It may not always work out this nicely, but it's
+ * better than with (NBuffers / num_nodes).
+ *
+ * Alternatively, we could "backtrack" - try with the large max_items,
+ * check how balanced it is, and if it's too imbalanced, try with a
+ * smaller one.
+ *
+ * We however want a simple scheme.
+ */
+
+ return num_items;
+}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_chunk_buffers == -1)
+ return -1;
+
+ return (buffer / numa_chunk_buffers) % numa_nodes;
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_interleave_memory(char *startptr, char *endptr,
+ Size mem_page_size, Size chunk_size,
+ int num_nodes)
+{
+ volatile uint64 touch pg_attribute_unused();
+ char *ptr = startptr;
+
+ /* chunk size has to be a multiple of memory page */
+ Assert((chunk_size % mem_page_size) == 0);
+
+ /*
+ * Walk the memory pages in the range, and determine the node for each
+ * one. We use numa_tonode_memory(), because then we can move a whole
+ * memory range to the node, we don't need to worry about individual pages
+ * like with numa_move_pages().
+ */
+ while (ptr < endptr)
+ {
+ /* We may have an incomplete chunk at the end. */
+ Size sz = Min(chunk_size, (endptr - ptr));
+
+ /*
+ * What NUMA node does this range belong to? Each chunk should go to
+ * the same NUMA node, in a round-robin manner.
+ */
+ int node = ((ptr - startptr) / chunk_size) % num_nodes;
+
+ /*
+ * Nope, we have the first buffer from the next memory page, and we'll
+ * set NUMA node for it (and all pages up to the next buffer). The
+ * buffer should align with the memory page, thanks to the
+ * buffer_align earlier.
+ */
+ Assert((int64) ptr % mem_page_size == 0);
+ Assert((sz % mem_page_size) == 0);
+
+ /*
+ * XXX no return value, to make this fail on error, has to use
+ * numa_set_strict
+ *
+ * XXX Should we still touch the memory first, like with numa_move_pages,
+ * or is that not necessary?
+ */
+ numa_tonode_memory(ptr, sz, node);
+
+ ptr += sz;
+ }
+
+ /* should have processed all chunks */
+ Assert(ptr == endptr);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 94db3e7c976..5922689fe5d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -685,6 +685,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
BufferDesc *bufHdr;
BufferTag tag;
uint32 buf_state;
+
Assert(BufferIsValid(recent_buffer));
ResourceOwnerEnlarge(CurrentResourceOwner);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9570087aa60 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 69b6a877dc9..c07de903f76 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -305,7 +305,7 @@ static const char *progname;
#define CPU_PINNING_RANDOM 1
#define CPU_PINNING_COLOCATED 2
-static int pinning_mode = CPU_PINNING_NONE;
+static int pinning_mode = CPU_PINNING_NONE;
#define WSEP '@' /* weight separator */
@@ -874,20 +874,20 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
*/
typedef struct cpu_generator_state
{
- int ncpus; /* number of CPUs available */
- int nitems; /* number of items in the queue */
- int *nthreads; /* number of threads for each CPU */
- int *nclients; /* number of processes for each CPU */
- int *items; /* queue of CPUs to pick from */
-} cpu_generator_state;
+ int ncpus; /* number of CPUs available */
+ int nitems; /* number of items in the queue */
+ int *nthreads; /* number of threads for each CPU */
+ int *nclients; /* number of processes for each CPU */
+ int *items; /* queue of CPUs to pick from */
+} cpu_generator_state;
static cpu_generator_state cpu_generator_init(int ncpus);
-static void cpu_generator_refill(cpu_generator_state *state);
-static void cpu_generator_reset(cpu_generator_state *state);
-static int cpu_generator_thread(cpu_generator_state *state);
-static int cpu_generator_client(cpu_generator_state *state, int thread_cpu);
-static void cpu_generator_print(cpu_generator_state *state);
-static bool cpu_generator_check(cpu_generator_state *state);
+static void cpu_generator_refill(cpu_generator_state * state);
+static void cpu_generator_reset(cpu_generator_state * state);
+static int cpu_generator_thread(cpu_generator_state * state);
+static int cpu_generator_client(cpu_generator_state * state, int thread_cpu);
+static void cpu_generator_print(cpu_generator_state * state);
+static bool cpu_generator_check(cpu_generator_state * state);
static void reset_pinning(TState *threads, int nthreads);
@@ -7422,7 +7422,7 @@ main(int argc, char **argv)
/* try to assign threads/clients to CPUs */
if (pinning_mode != CPU_PINNING_NONE)
{
- int nprocs = get_nprocs();
+ int nprocs = get_nprocs();
cpu_generator_state state = cpu_generator_init(nprocs);
retry:
@@ -7433,6 +7433,7 @@ retry:
for (i = 0; i < nthreads; i++)
{
TState *thread = &threads[i];
+
thread->cpu = cpu_generator_thread(&state);
}
@@ -7444,7 +7445,7 @@ retry:
while (true)
{
/* did we find any unassigned backend? */
- bool found = false;
+ bool found = false;
for (i = 0; i < nthreads; i++)
{
@@ -7678,10 +7679,10 @@ threadRun(void *arg)
/* determine PID of the backend, pin it to the same CPU */
for (int i = 0; i < nstate; i++)
{
- char *pid_str;
- pid_t pid;
+ char *pid_str;
+ pid_t pid;
- PGresult *res = PQexec(state[i].con, "select pg_backend_pid()");
+ PGresult *res = PQexec(state[i].con, "select pg_backend_pid()");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
pg_fatal("could not determine PID of the backend for client %d",
@@ -8184,7 +8185,7 @@ cpu_generator_init(int ncpus)
{
struct timeval tv;
- cpu_generator_state state;
+ cpu_generator_state state;
state.ncpus = ncpus;
@@ -8207,7 +8208,7 @@ cpu_generator_init(int ncpus)
}
static void
-cpu_generator_refill(cpu_generator_state *state)
+cpu_generator_refill(cpu_generator_state * state)
{
struct timeval tv;
@@ -8223,7 +8224,7 @@ cpu_generator_refill(cpu_generator_state *state)
}
static void
-cpu_generator_reset(cpu_generator_state *state)
+cpu_generator_reset(cpu_generator_state * state)
{
state->nitems = 0;
cpu_generator_refill(state);
@@ -8236,15 +8237,15 @@ cpu_generator_reset(cpu_generator_state *state)
}
static int
-cpu_generator_thread(cpu_generator_state *state)
+cpu_generator_thread(cpu_generator_state * state)
{
if (state->nitems == 0)
cpu_generator_refill(state);
while (true)
{
- int idx = lrand48() % state->nitems;
- int cpu = state->items[idx];
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
state->items[idx] = state->items[state->nitems - 1];
state->nitems--;
@@ -8256,10 +8257,10 @@ cpu_generator_thread(cpu_generator_state *state)
}
static int
-cpu_generator_client(cpu_generator_state *state, int thread_cpu)
+cpu_generator_client(cpu_generator_state * state, int thread_cpu)
{
- int min_clients;
- bool has_valid_cpus = false;
+ int min_clients;
+ bool has_valid_cpus = false;
for (int i = 0; i < state->nitems; i++)
{
@@ -8284,8 +8285,8 @@ cpu_generator_client(cpu_generator_state *state, int thread_cpu)
while (true)
{
- int idx = lrand48() % state->nitems;
- int cpu = state->items[idx];
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
if (cpu == thread_cpu)
continue;
@@ -8303,7 +8304,7 @@ cpu_generator_client(cpu_generator_state *state, int thread_cpu)
}
static void
-cpu_generator_print(cpu_generator_state *state)
+cpu_generator_print(cpu_generator_state * state)
{
for (int i = 0; i < state->ncpus; i++)
{
@@ -8312,10 +8313,10 @@ cpu_generator_print(cpu_generator_state *state)
}
static bool
-cpu_generator_check(cpu_generator_state *state)
+cpu_generator_check(cpu_generator_state * state)
{
- int min_count = INT_MAX,
- max_count = 0;
+ int min_count = INT_MAX,
+ max_count = 0;
for (int i = 0; i < state->ncpus; i++)
{
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..c257c8a1c20 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -319,6 +319,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
--
2.49.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-17 21:14 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-17 21:14 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/4/25 20:12, Tomas Vondra wrote:
> On 7/4/25 13:05, Jakub Wartak wrote:
>> ...
>>
>> 8. v1-0005 2x + /* if (numa_procs_interleave) */
>>
>> Ha! it's a TRAP! I've uncommented it because I wanted to try it out
>> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
>>
>> 2025-07-04 12:31:08.103 CEST [28754] LOG: starting PostgreSQL
>> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>> [..]
>> 2025-07-04 12:31:08.109 CEST [28754] LOG: io worker (PID 28755)
>> was terminated by signal 11: Segmentation fault
>> 2025-07-04 12:31:08.109 CEST [28754] LOG: terminating any other
>> active server processes
>> 2025-07-04 12:31:08.114 CEST [28754] LOG: shutting down because
>> "restart_after_crash" is off
>> 2025-07-04 12:31:08.116 CEST [28754] LOG: database system is shut down
>>
>> [New LWP 28755]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `postgres: io worker '.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>> at ./nptl/sem_waitcommon.c:136
>> 136 ./nptl/sem_waitcommon.c: No such file or directory.
>> (gdb) where
>> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>> at ./nptl/sem_waitcommon.c:136
>> #1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
>> #2 0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
>> ../src/backend/port/posix_sema.c:302
>> #3 0x0000556191970553 in InitAuxiliaryProcess () at
>> ../src/backend/storage/lmgr/proc.c:992
>> #4 0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
>> ../src/backend/postmaster/auxprocess.c:65
>> #5 0x0000556191940676 in IoWorkerMain (startup_data=<optimized
>> out>, startup_data_len=<optimized out>) at
>> ../src/backend/storage/aio/method_worker.c:393
>> #6 0x00005561918e8163 in postmaster_child_launch
>> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
>> startup_data=startup_data@entry=0x0,
>> startup_data_len=startup_data_len@entry=0,
>> client_sock=client_sock@entry=0x0) at
>> ../src/backend/postmaster/launch_backend.c:290
>> #7 0x00005561918ea09a in StartChildProcess
>> (type=type@entry=B_IO_WORKER) at
>> ../src/backend/postmaster/postmaster.c:3973
>> #8 0x00005561918ea308 in maybe_adjust_io_workers () at
>> ../src/backend/postmaster/postmaster.c:4404
>> [..]
>> (gdb) print *MyProc->sem
>> Cannot access memory at address 0x0
>>
>
> Yeah, good catch. I'll look into that next week.
>
I've been unable to reproduce this issue, but I'm not sure what settings
you actually used for this instance. Can you give me more details how to
reproduce this?
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-18 16:46 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-07-18 16:46 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Greg Burd <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
On 2025-07-17 23:11:16 +0200, Tomas Vondra wrote:
> Here's a v2 of the patch series, with a couple changes:
Not a deep look at the code, just a quick reply.
> * I changed the freelist partitioning scheme a little bit, based on the
> discussion in this thread. Instead of having a single "partition" per
> NUMA node, there's not a minimum number of partitions (set to 4). So
I assume s/not/now/?
> * There's now a patch partitioning clocksweep, using the same scheme as
> the freelists.
Nice!
> I came to the conclusion it doesn't make much sense to partition these
> things differently - I can't think of a reason why that would be
> advantageous, and it makes it easier to reason about.
Agreed.
> The clocksweep partitioning is somewhat harder, because it affects
> BgBufferSync() and related code. With the partitioning we now have
> multiple "clock hands" for different ranges of buffers, and the clock
> sweep needs to consider that. I modified BgBufferSync to simply loop
> through the ClockSweep partitions, and do a small cleanup for each.
That probably makes sense for now. It might need a bit of a larger adjustment
at some point, but ...
> * This new freelist/clocksweep partitioning scheme is however harder to
> disable. I now realize the GUC may quite do the trick, and there even is
> not a GUC for the clocksweep. I need to think about this, but I'm not
> even how feasible it'd be to have two separate GUCs (because of how
> these two pieces are intertwined). For now if you want to test without
> the partitioning, you need to skip the patch.
I think it's totally fair to enable/disable them at the same time. They're so
closely related, that I don't think it really makes sense to measure them
separately.
> I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
> the results are encouraging. For a read-only pgbench (2x shared buffers,
> within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
> bad considering the patch is more about consistency than raw throughput.
Personally I think an 1.18x improvement on a relatively small NUMA machine is
really rather awesome.
> For a read-write pgbench I however saw some strange drops/increases of
> throughput. I suspect this might be due to some thinko in the clocksweep
> partitioning, but I'll need to take a closer look.
Was that with pinning etc enabled or not?
> From c4d51ab87b92f9900e37d42cf74980e87b648a56 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <[email protected]>
> Date: Sun, 8 Jun 2025 18:53:12 +0200
> Subject: [PATCH v2 5/7] NUMA: clockweep partitioning
>
> @@ -475,13 +525,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
> /*
> * Nothing on the freelist, so run the "clock sweep" algorithm
> *
> - * XXX Should we also make this NUMA-aware, to only access buffers from
> - * the same NUMA node? That'd probably mean we need to make the clock
> - * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
> - * subset of buffers. But that also means each process could "sweep" only
> - * a fraction of buffers, even if the other buffers are better candidates
> - * for eviction. Would that also mean we'd have multiple bgwriters, one
> - * for each node, or would one bgwriter handle all of that?
> + * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
> + * buffers from a single partition, aligned with the NUMA node. That
> + * means it only accesses buffers from the same NUMA node.
> + *
> + * XXX That also means each process "sweeps" only a fraction of buffers,
> + * even if the other buffers are better candidates for eviction. Maybe
> + * there should be some logic to "steal" buffers from other freelists
> + * or other nodes?
I think we *definitely* need "stealing" from other clock sweeps, whenever
there's a meaningful imbalance between the different sweeps.
I don't think we need to be overly precise about it, a small imbalance won't
have that much of an effect. But clearly it doesn't make sense to say that one
backend can only fill buffers in the current partition, that'd lead to massive
performance issues in a lot of workloads.
The hardest thing probably is to make the logic for when to check foreign
clock sweeps cheap enough.
One way would be to do it whenever a sweep wraps around, that'd probably
amortize the cost sufficiently, and I don't think it'd be too imprecise, as
we'd have processed that set of buffers in a row without partitioning as
well. But it'd probably be too coarse when determining for how long to use a
foreign sweep instance. But we probably could address that by rechecking the
balanace more frequently when using a foreign partition.
Another way would be to have bgwriter manage this. Whenever it detects that
one ring is too far ahead, it could set a "avoid this partition" bit, which
would trigger backends that natively use that partition to switch to foreign
partitions that don't currently have that bit set. I suspect there's a
problem with that approach though, I worry that the amount of time that
bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
much imbalance.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-18 20:48 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-18 20:48 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Greg Burd <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
On 7/18/25 18:46, Andres Freund wrote:
> Hi,
>
> On 2025-07-17 23:11:16 +0200, Tomas Vondra wrote:
>> Here's a v2 of the patch series, with a couple changes:
>
> Not a deep look at the code, just a quick reply.
>
>
>> * I changed the freelist partitioning scheme a little bit, based on the
>> discussion in this thread. Instead of having a single "partition" per
>> NUMA node, there's not a minimum number of partitions (set to 4). So
>
> I assume s/not/now/?
>
Yes.
>
>> * There's now a patch partitioning clocksweep, using the same scheme as
>> the freelists.
>
> Nice!
>
>
>> I came to the conclusion it doesn't make much sense to partition these
>> things differently - I can't think of a reason why that would be
>> advantageous, and it makes it easier to reason about.
>
> Agreed.
>
>
>> The clocksweep partitioning is somewhat harder, because it affects
>> BgBufferSync() and related code. With the partitioning we now have
>> multiple "clock hands" for different ranges of buffers, and the clock
>> sweep needs to consider that. I modified BgBufferSync to simply loop
>> through the ClockSweep partitions, and do a small cleanup for each.
>
> That probably makes sense for now. It might need a bit of a larger
adjustment at some point, but ...
>
I couldn't think of something fundamentally better and not too complex.
I suspect we might want to use multiple bgwriters in the future, and
this scheme seems to be reasonably well suited for that too.
I'm also thinking about having some sort of "unified" partitioning
scheme for all the places partitioning shared buffers. Right now each of
the places does it on it's own, i.e. buff_init, freelist and clocksweep
all have their code splitting NBuffers into partitions. And it should
align. Because what would be the benefit if it didn't? But I guess
having three variants of the same code seems a bit pointless.
I think buff_init should build a common definition of buffer partitions,
and the remaining parts should use that as the source of truth ...
>
>> * This new freelist/clocksweep partitioning scheme is however harder to
>> disable. I now realize the GUC may quite do the trick, and there even is
>> not a GUC for the clocksweep. I need to think about this, but I'm not
>> even how feasible it'd be to have two separate GUCs (because of how
>> these two pieces are intertwined). For now if you want to test without
>> the partitioning, you need to skip the patch.
>
> I think it's totally fair to enable/disable them at the same time. They're so
> closely related, that I don't think it really makes sense to measure them
> separately.
>
Yeah, that's a fair point.
>
>> I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
>> the results are encouraging. For a read-only pgbench (2x shared buffers,
>> within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
>> bad considering the patch is more about consistency than raw throughput.
>
> Personally I think an 1.18x improvement on a relatively small NUMA machine is
> really rather awesome.
>
True, but I want to stress out it's just one quick (& simple test). Much
more testing is needed before I can make reliable claims.
>
>> For a read-write pgbench I however saw some strange drops/increases of
>> throughput. I suspect this might be due to some thinko in the clocksweep
>> partitioning, but I'll need to take a closer look.
>
> Was that with pinning etc enabled or not?
>
IIRC it was with everything enabled, except for numa_procs_pin (which
pins backend to NUMA node). I found that to actually harm performance in
some of the tests (even just read-only ones), resulting in uneven usage
of cores and lower throughput.
>
>
>> From c4d51ab87b92f9900e37d42cf74980e87b648a56 Mon Sep 17 00:00:00 2001
>> From: Tomas Vondra <[email protected]>
>> Date: Sun, 8 Jun 2025 18:53:12 +0200
>> Subject: [PATCH v2 5/7] NUMA: clockweep partitioning
>>
>
>
>> @@ -475,13 +525,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
>> /*
>> * Nothing on the freelist, so run the "clock sweep" algorithm
>> *
>> - * XXX Should we also make this NUMA-aware, to only access buffers from
>> - * the same NUMA node? That'd probably mean we need to make the clock
>> - * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
>> - * subset of buffers. But that also means each process could "sweep" only
>> - * a fraction of buffers, even if the other buffers are better candidates
>> - * for eviction. Would that also mean we'd have multiple bgwriters, one
>> - * for each node, or would one bgwriter handle all of that?
>> + * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
>> + * buffers from a single partition, aligned with the NUMA node. That
>> + * means it only accesses buffers from the same NUMA node.
>> + *
>> + * XXX That also means each process "sweeps" only a fraction of buffers,
>> + * even if the other buffers are better candidates for eviction. Maybe
>> + * there should be some logic to "steal" buffers from other freelists
>> + * or other nodes?
>
> I think we *definitely* need "stealing" from other clock sweeps, whenever
> there's a meaningful imbalance between the different sweeps.
>
> I don't think we need to be overly precise about it, a small imbalance won't
> have that much of an effect. But clearly it doesn't make sense to say that one
> backend can only fill buffers in the current partition, that'd lead to massive
> performance issues in a lot of workloads.
>
Agreed.
> The hardest thing probably is to make the logic for when to check foreign
> clock sweeps cheap enough.
>
> One way would be to do it whenever a sweep wraps around, that'd probably
> amortize the cost sufficiently, and I don't think it'd be too imprecise, as
> we'd have processed that set of buffers in a row without partitioning as
> well. But it'd probably be too coarse when determining for how long to use a
> foreign sweep instance. But we probably could address that by rechecking the
> balanace more frequently when using a foreign partition.
>
What you mean by "it"? What would happen after a sweep wraps around?
> Another way would be to have bgwriter manage this. Whenever it detects that
> one ring is too far ahead, it could set a "avoid this partition" bit, which
> would trigger backends that natively use that partition to switch to foreign
> partitions that don't currently have that bit set. I suspect there's a
> problem with that approach though, I worry that the amount of time that
> bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
> much imbalance.
>
I'm afraid having hard "avoid" flags would lead to sudden and unexpected
changes in performance as we enable/disable partitions. I think a good
solution should "smooth it out" somehow, e.g. by not having a true/false
flag, but having some sort of "preference" factor with values between
(0.0, 1.0) which says how much we should use that partition.
I was imagining something like this:
Say we know the number of buffers allocated for each partition (in the
last round), and we (or rather the BgBufferSync) calculate:
coefficient = 1.0 - (nallocated_partition / nallocated)
and then use that to "correct" which partition to allocate buffers from.
Or maybe just watch how far from the "fair share" we were in the last
interval, and gradually increase/decrease the "partition preference"
which would say how often we need to "steal" from other partitions.
E.g. we find nallocated_partition is 2x the fair share, i.e.
nallocated_partition / (nallocated / nparts) = 2.0
Then we say 25% of the time look at some other partition, to "cut" the
imbalance in half. And then repeat that in the next cycle, etc.
So a process would look at it's "home partition" by default, but it's
"roll a dice" first and if above the calculated probability it'd pick
some other partition instead (this would need to be done so that it gets
balanced overall).
If the bgwriter interval is too long, maybe the recalculation could be
triggered regularly after any of the clocksweeps wraps around, or after
some number of allocations, or something like that.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-18 21:03 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Andres Freund @ 2025-07-18 21:03 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Greg Burd <[email protected]>; Ashutosh Bapat <[email protected]>; PostgreSQL Hackers <[email protected]>; Dmitry Dolgov <[email protected]>
Hi,
On 2025-07-18 22:48:00 +0200, Tomas Vondra wrote:
> On 7/18/25 18:46, Andres Freund wrote:
> >> For a read-write pgbench I however saw some strange drops/increases of
> >> throughput. I suspect this might be due to some thinko in the clocksweep
> >> partitioning, but I'll need to take a closer look.
> >
> > Was that with pinning etc enabled or not?
> >
>
> IIRC it was with everything enabled, except for numa_procs_pin (which
> pins backend to NUMA node). I found that to actually harm performance in
> some of the tests (even just read-only ones), resulting in uneven usage
> of cores and lower throughput.
FWIW, I really doubt that something like numa_procs_pin is viable outside of
very narrow niches until we have a *lot* more infrastructure in place. Like PG
would need to be threaded, we'd need a separation between thread and
connection and an executor that'd allow us to switch from working on one query
to working on another query.
> > The hardest thing probably is to make the logic for when to check foreign
> > clock sweeps cheap enough.
> >
> > One way would be to do it whenever a sweep wraps around, that'd probably
> > amortize the cost sufficiently, and I don't think it'd be too imprecise, as
> > we'd have processed that set of buffers in a row without partitioning as
> > well. But it'd probably be too coarse when determining for how long to use a
> > foreign sweep instance. But we probably could address that by rechecking the
> > balanace more frequently when using a foreign partition.
> >
>
> What you mean by "it"?
it := Considering switching back from using a "foreign" clock sweep instance
whenever the sweep wraps around.
> What would happen after a sweep wraps around?
The scenario I'm worried about is this:
1) a bunch of backends read buffers on numa node A, using the local clock
sweep instance
2) due to all of that activity, the clock sweep advances much faster than the
clock sweep for numa node B
3) the clock sweep on A wraps around, we discover the imbalance, and all the
backend switch to scanning on numa node B, moving that clock sweep ahead
much more aggressively
4) clock sweep on B wraps around, there's imbalance the other way round now,
so they all switch back to A
> > Another way would be to have bgwriter manage this. Whenever it detects that
> > one ring is too far ahead, it could set a "avoid this partition" bit, which
> > would trigger backends that natively use that partition to switch to foreign
> > partitions that don't currently have that bit set. I suspect there's a
> > problem with that approach though, I worry that the amount of time that
> > bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
> > much imbalance.
> >
>
> I'm afraid having hard "avoid" flags would lead to sudden and unexpected
> changes in performance as we enable/disable partitions. I think a good
> solution should "smooth it out" somehow, e.g. by not having a true/false
> flag, but having some sort of "preference" factor with values between
> (0.0, 1.0) which says how much we should use that partition.
Yea, I think that's a fair worry.
> I was imagining something like this:
>
> Say we know the number of buffers allocated for each partition (in the
> last round), and we (or rather the BgBufferSync) calculate:
>
> coefficient = 1.0 - (nallocated_partition / nallocated)
>
> and then use that to "correct" which partition to allocate buffers from.
> Or maybe just watch how far from the "fair share" we were in the last
> interval, and gradually increase/decrease the "partition preference"
> which would say how often we need to "steal" from other partitions.
>
> E.g. we find nallocated_partition is 2x the fair share, i.e.
>
> nallocated_partition / (nallocated / nparts) = 2.0
>
> Then we say 25% of the time look at some other partition, to "cut" the
> imbalance in half. And then repeat that in the next cycle, etc.
>
> So a process would look at it's "home partition" by default, but it's
> "roll a dice" first and if above the calculated probability it'd pick
> some other partition instead (this would need to be done so that it gets
> balanced overall).
That does sound reasonable.
> If the bgwriter interval is too long, maybe the recalculation could be
> triggered regularly after any of the clocksweeps wraps around, or after
> some number of allocations, or something like that.
I'm pretty sure the bgwriter might not be often enough and not predictably
frequently running for that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-25 10:27 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-07-25 10:27 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Thu, Jul 17, 2025 at 11:15 PM Tomas Vondra <[email protected]> wrote:
>
> On 7/4/25 20:12, Tomas Vondra wrote:
> > On 7/4/25 13:05, Jakub Wartak wrote:
> >> ...
> >>
> >> 8. v1-0005 2x + /* if (numa_procs_interleave) */
> >>
> >> Ha! it's a TRAP! I've uncommented it because I wanted to try it out
> >> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
> >>
> >> 2025-07-04 12:31:08.103 CEST [28754] LOG: starting PostgreSQL
> >> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
> >> [..]
> >> 2025-07-04 12:31:08.109 CEST [28754] LOG: io worker (PID 28755)
> >> was terminated by signal 11: Segmentation fault
> >> 2025-07-04 12:31:08.109 CEST [28754] LOG: terminating any other
> >> active server processes
> >> 2025-07-04 12:31:08.114 CEST [28754] LOG: shutting down because
> >> "restart_after_crash" is off
> >> 2025-07-04 12:31:08.116 CEST [28754] LOG: database system is shut down
> >>
> >> [New LWP 28755]
> >> [Thread debugging using libthread_db enabled]
> >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >> Core was generated by `postgres: io worker '.
> >> Program terminated with signal SIGSEGV, Segmentation fault.
> >> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> >> at ./nptl/sem_waitcommon.c:136
> >> 136 ./nptl/sem_waitcommon.c: No such file or directory.
> >> (gdb) where
> >> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> >> at ./nptl/sem_waitcommon.c:136
> >> #1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
> >> #2 0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
> >> ../src/backend/port/posix_sema.c:302
> >> #3 0x0000556191970553 in InitAuxiliaryProcess () at
> >> ../src/backend/storage/lmgr/proc.c:992
> >> #4 0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
> >> ../src/backend/postmaster/auxprocess.c:65
> >> #5 0x0000556191940676 in IoWorkerMain (startup_data=<optimized
> >> out>, startup_data_len=<optimized out>) at
> >> ../src/backend/storage/aio/method_worker.c:393
> >> #6 0x00005561918e8163 in postmaster_child_launch
> >> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
> >> startup_data=startup_data@entry=0x0,
> >> startup_data_len=startup_data_len@entry=0,
> >> client_sock=client_sock@entry=0x0) at
> >> ../src/backend/postmaster/launch_backend.c:290
> >> #7 0x00005561918ea09a in StartChildProcess
> >> (type=type@entry=B_IO_WORKER) at
> >> ../src/backend/postmaster/postmaster.c:3973
> >> #8 0x00005561918ea308 in maybe_adjust_io_workers () at
> >> ../src/backend/postmaster/postmaster.c:4404
> >> [..]
> >> (gdb) print *MyProc->sem
> >> Cannot access memory at address 0x0
> >>
> >
> > Yeah, good catch. I'll look into that next week.
> >
>
> I've been unable to reproduce this issue, but I'm not sure what settings
> you actually used for this instance. Can you give me more details how to
> reproduce this?
Better late than never, well feel free to partially ignore me, i've
missed that it is known issue as per FIXME there, but I would just rip
out that commented out `if(numa_proc_interleave)` from
FastPathLockShmemSize() and PGProcShmemSize() unless you want to save
those memory pages of course (in case of no-NUMA). If you do want to
save those pages I think we have problem:
For complete picture, steps:
1. patch -p1 < v2-0001-NUMA-interleaving-buffers.patch
2. patch -p1 < v2-0006-NUMA-interleave-PGPROC-entries.patch
BTW the pgbench accidentinal ident is still there (part of v2-0001 patch))
14 out of 14 hunks FAILED -- saving rejects to file
src/bin/pgbench/pgbench.c.rej
3. As I'm just applying 0001 and 0006, I've got two simple rejects,
but fixed it (due to not applying missing numa_ freelist patches).
That's intentional on my part, because I wanted to play just with
those two.
4. Then I uncomment those two "if (numa_procs_interleave)" related for
optional memory shm initialization - add_size() and so on (that have
XXX comment above that it is causing bootstrap issues)
5. initdb with numa_procs_interleave=on, huge_pages = on (!), start, it is ok
6. restart with numa_procs_interleave=off, which gets me to every bg
worker crashing e.g.:
(gdb) where
#0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0) at
./nptl/sem_waitcommon.c:136
#1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
#2 0x0000563e2d6e4d5c in PGSemaphoreReset (sema=0x0) at
../src/backend/port/posix_sema.c:302
#3 0x0000563e2d774d93 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:995
#4 0x0000563e2d6e9252 in AuxiliaryProcessMainCommon () at
../src/backend/postmaster/auxprocess.c:65
#5 0x0000563e2d6eb683 in CheckpointerMain (startup_data=<optimized
out>, startup_data_len=<optimized out>) at
../src/backend/postmaster/checkpointer.c:190
#6 0x0000563e2d6ec363 in postmaster_child_launch
(child_type=child_type@entry=B_CHECKPOINTER, child_slot=249,
startup_data=startup_data@entry=0x0,
startup_data_len=startup_data_len@entry=0,
client_sock=client_sock@entry=0x0) at
../src/backend/postmaster/launch_backend.c:290
#7 0x0000563e2d6ee29a in StartChildProcess
(type=type@entry=B_CHECKPOINTER) at
../src/backend/postmaster/postmaster.c:3973
#8 0x0000563e2d6f17a6 in PostmasterMain (argc=argc@entry=3,
argv=argv@entry=0x563e377cc0e0) at
../src/backend/postmaster/postmaster.c:1386
#9 0x0000563e2d4948fc in main (argc=3, argv=0x563e377cc0e0) at
../src/backend/main/main.c:231
notice sema=0x0, because:
#3 0x000056050928cd93 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:995
995 PGSemaphoreReset(MyProc->sem);
(gdb) print MyProc
$1 = (PGPROC *) 0x7f09a0c013b0
(gdb) print MyProc->sem
$2 = (PGSemaphore) 0x0
or with printfs:
2025-07-25 11:17:23.683 CEST [21772] LOG: in InitProcGlobal
PGPROC=0x7f9de827b880 requestSize=148770
// after proc && ptr manipulation:
2025-07-25 11:17:23.683 CEST [21772] LOG: in InitProcGlobal
PGPROC=0x7f9de827bdf0 requestSize=148770 procs=0x7f9de827b880
ptr=0x7f9de827bdf0
[..initialization of aux PGPROCs i=0.., still fromInitProcGlobal(),
each gets proper sem allocated as one would expect:]
[..for i loop:]
2025-07-25 11:17:23.689 CEST [21772] LOG: i=136 ,
proc=0x7f9de8600000, proc->sem=0x7f9da4e04438
2025-07-25 11:17:23.689 CEST [21772] LOG: i=137 ,
proc=0x7f9de8600348, proc->sem=0x7f9da4e044b8
2025-07-25 11:17:23.689 CEST [21772] LOG: i=138 ,
proc=0x7f9de8600690, proc->sem=0x7f9da4e04538
[..but then in the children codepaths, out of the blue in
InitAuxilaryProcess the whole MyProc looks like it would memsetted to
zeros:]
2025-07-25 11:17:23.693 CEST [21784] LOG: auxiliary process using
MyProc=0x7f9de8600000 auxproc=0x7f9de8600000 proctype=0
MyProcPid=21784 MyProc->sem=(nil)
above got pgproc slot i=136 with addr 0x7f9de8600000 and later that
auxiliary is launched but somehow something NULLified ->sem there
(according to gdb , everything is zero there)
7. Original patch v2-0006 (with commented out 2x if
numa_procs_interleave), behaves OK, so in my case here with 1x NUMA
node that gives add_size(.., 1+1 * 2MB)=4MB
2025-07-25 11:38:54.131 CEST [23939] LOG: in InitProcGlobal
PGPROC=0x7f25cbe7b880 requestSize=4343074
2025-07-25 11:38:54.132 CEST [23939] LOG: in InitProcGlobal
PGPROC=0x7f25cbe7bdf0 requestSize=4343074 procs=0x7f25cbe7b880
ptr=0x7f25cbe7bdf0
so something is zeroing out all those MyProc structures apparently on
startup (probably due to some wrong alignment maybe somewhere ?) I was
thinking about trapping via mprotect() this single i=136
0x7f9de8600000 PGPROC to see what is resetting it, but oh well,
mprotect() works only on whole pages...
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-25 10:51 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-25 10:51 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/25/25 12:27, Jakub Wartak wrote:
> On Thu, Jul 17, 2025 at 11:15 PM Tomas Vondra <[email protected]> wrote:
>>
>> On 7/4/25 20:12, Tomas Vondra wrote:
>>> On 7/4/25 13:05, Jakub Wartak wrote:
>>>> ...
>>>>
>>>> 8. v1-0005 2x + /* if (numa_procs_interleave) */
>>>>
>>>> Ha! it's a TRAP! I've uncommented it because I wanted to try it out
>>>> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
>>>>
>>>> 2025-07-04 12:31:08.103 CEST [28754] LOG: starting PostgreSQL
>>>> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>>>> [..]
>>>> 2025-07-04 12:31:08.109 CEST [28754] LOG: io worker (PID 28755)
>>>> was terminated by signal 11: Segmentation fault
>>>> 2025-07-04 12:31:08.109 CEST [28754] LOG: terminating any other
>>>> active server processes
>>>> 2025-07-04 12:31:08.114 CEST [28754] LOG: shutting down because
>>>> "restart_after_crash" is off
>>>> 2025-07-04 12:31:08.116 CEST [28754] LOG: database system is shut down
>>>>
>>>> [New LWP 28755]
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `postgres: io worker '.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>>>> at ./nptl/sem_waitcommon.c:136
>>>> 136 ./nptl/sem_waitcommon.c: No such file or directory.
>>>> (gdb) where
>>>> #0 __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>>>> at ./nptl/sem_waitcommon.c:136
>>>> #1 __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
>>>> #2 0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
>>>> ../src/backend/port/posix_sema.c:302
>>>> #3 0x0000556191970553 in InitAuxiliaryProcess () at
>>>> ../src/backend/storage/lmgr/proc.c:992
>>>> #4 0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
>>>> ../src/backend/postmaster/auxprocess.c:65
>>>> #5 0x0000556191940676 in IoWorkerMain (startup_data=<optimized
>>>> out>, startup_data_len=<optimized out>) at
>>>> ../src/backend/storage/aio/method_worker.c:393
>>>> #6 0x00005561918e8163 in postmaster_child_launch
>>>> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
>>>> startup_data=startup_data@entry=0x0,
>>>> startup_data_len=startup_data_len@entry=0,
>>>> client_sock=client_sock@entry=0x0) at
>>>> ../src/backend/postmaster/launch_backend.c:290
>>>> #7 0x00005561918ea09a in StartChildProcess
>>>> (type=type@entry=B_IO_WORKER) at
>>>> ../src/backend/postmaster/postmaster.c:3973
>>>> #8 0x00005561918ea308 in maybe_adjust_io_workers () at
>>>> ../src/backend/postmaster/postmaster.c:4404
>>>> [..]
>>>> (gdb) print *MyProc->sem
>>>> Cannot access memory at address 0x0
>>>>
>>>
>>> Yeah, good catch. I'll look into that next week.
>>>
>>
>> I've been unable to reproduce this issue, but I'm not sure what settings
>> you actually used for this instance. Can you give me more details how to
>> reproduce this?
>
> Better late than never, well feel free to partially ignore me, i've
> missed that it is known issue as per FIXME there, but I would just rip
> out that commented out `if(numa_proc_interleave)` from
> FastPathLockShmemSize() and PGProcShmemSize() unless you want to save
> those memory pages of course (in case of no-NUMA). If you do want to
> save those pages I think we have problem:
>
> For complete picture, steps:
>
> 1. patch -p1 < v2-0001-NUMA-interleaving-buffers.patch
> 2. patch -p1 < v2-0006-NUMA-interleave-PGPROC-entries.patch
>
> BTW the pgbench accidentinal ident is still there (part of v2-0001 patch))
> 14 out of 14 hunks FAILED -- saving rejects to file
> src/bin/pgbench/pgbench.c.rej
>
> 3. As I'm just applying 0001 and 0006, I've got two simple rejects,
> but fixed it (due to not applying missing numa_ freelist patches).
> That's intentional on my part, because I wanted to play just with
> those two.
>
> 4. Then I uncomment those two "if (numa_procs_interleave)" related for
> optional memory shm initialization - add_size() and so on (that have
> XXX comment above that it is causing bootstrap issues)
>
Ah, I didn't realize you uncommented these "if" conditions. In that case
the crash is not very surprising, because the actual initialization in
InitProcGlobal ignores the GUCs and just assumes it's enabled. But
without the extra padding that likely messes up something. Or something
allocated later "overwrites" the some of the memory.
I need to clean this up, to actually consider the GUC properly.
FWIW I do have a new patch version that I plan to share in a day or two,
once I get some numbers. It didn't change this particular part, though,
it's more about the buffers/freelists/clocksweep. I'll work on PGPROC
next, I think.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-28 14:19 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-28 14:19 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
Here's a somewhat cleaned up v3 of this patch series, with various
improvements and a lot of cleanup. Still WIP, but I hope it resolves the
various crashes reported for v2, but it still requires --with-libnuma
(it won't build without it).
I'm aware there's an ongoing discussion about removing the freelists,
and changing the clocksweep in some way. If that happens, the relevant
parts of this series will need some adjustment, of course. I haven't
looked into that yet, I plan to review those patches soon.
main changes in v3
------------------
1) I've introduced "registry" of the buffer partitions (imagine a small
array of structs), serving as a source of truth for places that need
info about the partitions (range of buffers, ...).
With v2 there was no "shared definition" - the shared buffers, freelist
and clocksweep did their own thing. But per the discussion it doesn't
really make much sense for to partition buffers in different ways.
So in v3 the 0001 patch defines the partitions, records them in shared
memory (in a small array), and the later parts just reuse this.
I also added a pg_buffercache_partitions() listing the partitions, with
first/last buffer, etc. The freelist/clocksweep patches add additional
information.
2) The PGPROC part introduces a similar registry, even though there are
no other patches building on this. But it seemed useful to have a clear
place recording this info.
There's also a view pg_buffercache_pgproc. The pg_buffercache location
is a bit bogus - it has nothing to do with buffers, but it was good
enough for now.
3) The PGPROC partitioning is reworked and should fix the crash with the
GUC set to "off".
4) This still doesn't do anything about "balancing" the clocksweep. I
have some ideas how to do that, I'll work on that next.
simple benchmark
----------------
I did a simple benchmark, measuring pgbench throughput with scale still
fitting into RAM, but much larger (~2x) than shared buffers. See the
attached test script, testing builds with more and more of the patches.
I'm attaching results from two different machines (the "usual" 2P xeon
and also a much larger cloud instance with EPYC/Genoa) - both the raw
CSV files, with average tps and percentiles, and PDFs. The PDFs also
have a comparison either to the "preceding" build (right side), or to
master (below the table).
There's results for the three "pgbench pinning" strategies, and that can
have pretty significant impact (colocated generally performs much better
than either "none" or "random").
For the "bigger" machine (wiuth 176 cores) the incremental results look
like this (for pinning=none, i.e. regular pgbench):
mode s_b buffers localal no-tail freelist sweep pgproc pinning
====================================================================
prepared 16GB 99% 101% 100% 103% 111% 99% 102%
32GB 98% 102% 99% 103% 107% 101% 112%
8GB 97% 102% 100% 102% 101% 101% 106%
--------------------------------------------------------------------
simple 16GB 100% 100% 99% 105% 108% 99% 108%
32GB 98% 101% 100% 103% 100% 101% 97%
8GB 100% 100% 101% 99% 100% 104% 104%
The way I read this is that the first three patches have about no impact
on throughput. Then freelist partitioning and (especially) clocksweep
partitioning can help quite a bit. pgproc is again close to ~0%, and
PGPROC pinning can help again (but this part is merely experimental).
For the xeon the differences (in either direction) are much smaller, so
I'm not going to post it here. It's in the PDF, though.
I think this looks reasonable. The way I see this patch series is not
about improving peak throughput, but more about reducing imbalance and
making the behavior more consistent.
The results are more a confirmation there's not some sort of massive
overhead, somewhere. But I'll get to this in a minute.
To quantify this kind of improvement, I think we'll need tests that
intentionally cause (or try to) imbalance. If you have ideas for such
tests, let me know.
overhead of partitioning calculation
------------------------------------
Regarding the "overhead", while the results look mostly OK, I think
we'll need to rethink the partitioning scheme - particularly how the
partition size is calculated. The current scheme has to use %, which can
be somewhat expensive.
The 0001 patch calculates a "chunk size", which is the smallest number
of buffers it can "assign" to a NUMA node. This depends on how many
buffer descriptors fit onto a single memory page, and it's either 512KB
(with 4KB pages), or 256MB (with 2MB huge pages). And then each NUMA
node gets multiple chunks, to cover shared_buffers/num_nodes. But this
can be an arbitrary number - it minimizes the imbalance, but it also
forces the use of % and / in the formulas.
AFAIK if we required the partitions to be 2^k multiples of the chunk
size, we could switch to using shifts and masking. Which is supposed to
be much faster. But I haven't measured this, and the cost is that some
of the nodes could get much less memory. Maybe that's fine.
reserving number of huge pages
------------------------------
The other thing I realized is that partitioning buffers with huge pages
is quite tricky, and can easily lead to SIGBUS when accessing the memory
later. The crashes I saw happen like this:
1) figure # of pages needed (using shared_memory_size_in_huge_pages)
This can be 16828 for shared_buffers=32GB.
2) make sure there's enough huge pages
echo 16828 > /proc/sys/vm/nr_hugepages
3) start postgres - everything seems to works just fine
4) query pg_buffercache_numa - triggers SIGBUS accessing memory for a
valid buffer (usually ~2GB from the end)
It took me ages to realize what's happening, but it's very simple. The
nr_hugepages is a global limit, but it's also translated into limits for
each NUMA node. So when you write 16828 to it, in a 4-node system each
node gets 1/4 of that. See
$ numastat -cm
Then we do the mmap(), and everything looks great, because there really
is enough huge pages and the system can allocate memory from any NUMA
node it needs.
And then we come around, and do the numa_tonode_memory(). And that's
where the issues start, because AFAIK this does not check the per-node
limit of huge pages in any way. It just appears to work. And then later,
when we finally touch the buffer, it tries to actually allocate the
memory on the node, and realizes there's not enough huge pages. And
triggers the SIGBUS.
You may ask why the per-node limit is too low. We still need just
shared_memory_size_in_huge_pages, right? And if we were partitioning the
whole memory segment, that'd be true. But we only to that for shared
buffers, and there's a lot of other shared memory - could be 1-2GB or
so, depending on the configuration.
And this gets placed on one of the nodes, and it counts against the
limit on that particular node. And so it doesn't have enough huge pages
to back the partition of shared buffers.
The only way around this I found is by inflating the number of huge
pages, significantly above the shared_memory_size_in_huge_pages value.
Just to make sure the nodes get enough huge pages.
I don't know what to do about this. It's quite annoying. If we only used
huge pages for the partitioned parts, this wouldn't be a problem.
I also realize this can be used to make sure the memory is balanced on
NUMA systems. Because if you set nr_hugepages, the kernel will ensure
the shared memory is distributed on all the nodes.
It won't have the benefits of "coordinating" the buffers and buffer
descriptors, and so on. But it will be balanced.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v3-0007-NUMA-pin-backends-to-NUMA-nodes.patch (3.5K, ../../[email protected]/2-v3-0007-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From 3b3a929b007205c46b3c193cf38e3edf71084af7 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v3 7/7] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 11259151a7d..dbb4cbb1bfa 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -766,6 +766,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ee4684d1b8..3f88659b49f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7b718760248..862341e137e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2156,6 +2156,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index cdeee8dccba..a97741c6707 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v3-0006-NUMA-interleave-PGPROC-entries.patch (46.8K, ../../[email protected]/3-v3-0006-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From 5addb5973ce571debebf07b17adc07eb828a48ee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v3 6/7] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 -
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 63 +-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 565 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 14 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 64 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index b7d8ea45ed7..c48950a9d3b 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -23,3 +23,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 5169655ae78..22396f36c09 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/rel.h"
@@ -28,6 +29,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -102,6 +104,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -905,3 +908,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..087279a6a8e 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 5b65a855b29..fb52039e1a6 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -500,8 +500,6 @@ buffer_partitions_prepare(void)
if (numa_nodes < 1)
numa_nodes = 1;
- elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
-
/*
* XXX A bit weird. Do we need to worry about postmaster? Could this even
* run outside postmaster? I don't think so.
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index ff02dc8e00b..d8d602f0a4e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -416,7 +416,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index bf987aed8d3..3e86e4ca2ae 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1622,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1739,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2223,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2297,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2498,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2724,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2755,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2857,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3019,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3060,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3188,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3231,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3300,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3402,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3467,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3522,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3568,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3597,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3628,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3669,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3732,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3798,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 62f3471448e..c84a2a5f1bc 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..11259151a7d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -90,6 +98,31 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static Size get_memory_page_size(void); /* XXX duplicate with bufi_init.c */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +133,63 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -129,6 +214,25 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -140,6 +244,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -191,7 +298,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -205,6 +312,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -241,19 +350,115 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partitoion
+ * per NUMA node.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += (numa_nodes * sizeof(PGProcPartition));
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /*
+ * Now initialize the PGPROC partition registry with a single
+ * partition for all the procs.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += sizeof(PGProcPartition);
+
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +491,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,15 +639,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +705,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2296,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2371,222 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(LOG, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(LOG, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (numa_procs_interleave && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a11bc71a386..6ee4684d1b8 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0552ed62cc7..7b718760248 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2146,6 +2146,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 66baf2bf33e..cdeee8dccba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..d2d269941fc 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -443,8 +448,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a38dd8d6242..5595cd48eee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1877,6 +1877,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.50.1
[text/x-patch] v3-0005-NUMA-clockweep-partitioning.patch (39.4K, ../../[email protected]/4-v3-0005-NUMA-clockweep-partitioning.patch)
download | inline diff:
From 2bfd4a824b12e9a865c5ef0a8ed33e215fb1b698 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v3 5/7] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/bufmgr.c | 476 ++++++++++--------
src/backend/storage/buffer/freelist.c | 224 +++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 478 insertions(+), 261 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 3871c261528..b7d8ea45ed7 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -12,7 +12,10 @@ LANGUAGE C PARALLEL SAFE;
-- Create a view for convenient access.
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
- (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer, buffers_consumed bigint, buffers_remain bigint, buffers_free bigint);
+ (partition integer,
+ numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer,
+ buffers_consumed bigint, buffers_remain bigint, buffers_free bigint,
+ complete_passes bigint, buffer_allocs bigint, next_victim_buffer integer);
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 668ada8c47b..5169655ae78 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "buffers_free",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "complete_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "buffer_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "next_victim_buffer",
+ INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,6 +849,10 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
+ uint32 complete_passes,
+ buffer_allocs,
+ next_victim_buffer;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -850,7 +860,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
- &buffers_free);
+ &buffers_free, &complete_passes,
+ &buffer_allocs, &next_victim_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -876,6 +887,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffers_free);
nulls[7] = false;
+ values[8] = Int32GetDatum(complete_passes);
+ nulls[8] = false;
+
+ values[9] = Int32GetDatum(buffer_allocs);
+ nulls[9] = false;
+
+ values[10] = Int32GetDatum(next_victim_buffer);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5922689fe5d..bd007c1c621 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3587,6 +3587,23 @@ BufferSync(int flags)
TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_scan);
}
+/*
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
+ *
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
+ *
+ * XXX might be better to have a per-partition struct with all the info
+ */
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
/*
* BgBufferSync -- Write out some dirty buffers in the pool.
*
@@ -3602,55 +3619,24 @@ bool
BgBufferSync(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
- int strategy_buf_id;
- uint32 strategy_passes;
uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
-
- /* Moving averages of allocation rate and clean-buffer density */
- static float smoothed_alloc = 0;
- static float smoothed_density = 10.0;
-
- /* Potentially these could be tunables, but for now, not */
- float smoothing_samples = 16;
- float scan_whole_pool_milliseconds = 120000.0;
-
- /* Used to compute how far we scan ahead */
- long strategy_delta;
- int bufs_to_lap;
- int bufs_ahead;
- float scans_per_alloc;
- int reusable_buffers_est;
- int upcoming_alloc_est;
- int min_scan_buffers;
-
- /* Variables for the scanning loop proper */
- int num_to_scan;
- int num_written;
- int reusable_buffers;
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
- /* Variables for final smoothed_density update */
- long new_strategy_delta;
- uint32 new_recent_alloc;
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
- /*
- * Find out where the freelist clock sweep currently is, and how many
- * buffer allocations have happened since our last call.
- */
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
/* Report buffer alloc counts to pgstat */
PendingBgWriterStats.buf_alloc += recent_alloc;
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -3663,223 +3649,285 @@ BgBufferSync(WritebackContext *wb_context)
}
/*
- * Compute strategy_delta = how many buffers have been scanned by the
- * clock sweep since last time. If first time through, assume none. Then
- * see if we are still ahead of the clock sweep, and if so, how many
- * buffers we could scan before we'd catch up with it and "lap" it. Note:
- * weird-looking coding of xxx_passes comparisons are to avoid bogus
- * behavior when the passes counts wrap around.
- */
- if (saved_info_valid)
- {
- int32 passes_delta = strategy_passes - prev_strategy_passes;
-
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* info obtained from freelist.c */
+ int strategy_buf_id;
+ uint32 strategy_passes;
+
+ /* Moving averages of allocation rate and clean-buffer density */
+ static float smoothed_alloc = 0;
+ static float smoothed_density = 10.0;
+
+ /* Potentially these could be tunables, but for now, not */
+ float smoothing_samples = 16;
+ float scan_whole_pool_milliseconds = 120000.0;
+
+ /* Used to compute how far we scan ahead */
+ long strategy_delta;
+ int bufs_to_lap;
+ int bufs_ahead;
+ float scans_per_alloc;
+ int reusable_buffers_est;
+ int upcoming_alloc_est;
+ int min_scan_buffers;
+
+ /* Variables for the scanning loop proper */
+ int num_to_scan;
+ int num_written;
+ int reusable_buffers;
+
+ /* Variables for final smoothed_density update */
+ long new_strategy_delta;
+ uint32 new_recent_alloc;
+
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
- Assert(strategy_delta >= 0);
+ /*
+ * Find out where the freelist clock sweep currently is, and how many
+ * buffer allocations have happened since our last call.
+ */
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
- if ((int32) (next_passes - strategy_passes) > 0)
+ /*
+ * Compute strategy_delta = how many buffers have been scanned by the
+ * clock sweep since last time. If first time through, assume none.
+ * Then see if we are still ahead of the clock sweep, and if so, how
+ * many buffers we could scan before we'd catch up with it and "lap"
+ * it. Note: weird-looking coding of xxx_passes comparisons are to
+ * avoid bogus behavior when the passes counts wrap around.
+ */
+ if (saved_info_valid)
{
- /* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
+
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
+
+ Assert(strategy_delta >= 0);
+
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
+ {
+ /* we're one pass ahead of the strategy point */
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
#endif
- }
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
- {
- /* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ }
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
+ {
+ /* on same pass, but ahead or at least not behind */
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
+#ifdef BGW_DEBUG
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
+#endif
+ }
+ else
+ {
+ /*
+ * We're behind, so skip forward to the strategy point and
+ * start cleaning from there.
+ */
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta);
#endif
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
+ }
}
else
{
/*
- * We're behind, so skip forward to the strategy point and start
- * cleaning from there.
+ * Initializing at startup or after LRU scanning had been off.
+ * Always start at the strategy point.
*/
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta);
+ elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
+ strategy_passes, strategy_buf_id);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ strategy_delta = 0;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
- }
- else
- {
- /*
- * Initializing at startup or after LRU scanning had been off. Always
- * start at the strategy point.
- */
-#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
- strategy_passes, strategy_buf_id);
-#endif
- strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
- }
- /* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ /* Update saved info for next time */
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* FIXME has to happen after all partitions */
+ /* saved_info_valid = true; */
- /*
- * Compute how many buffers had to be scanned for each new allocation, ie,
- * 1/density of reusable buffers, and track a moving average of that.
- *
- * If the strategy point didn't move, we don't update the density estimate
- */
- if (strategy_delta > 0 && recent_alloc > 0)
- {
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
- }
+ /*
+ * Compute how many buffers had to be scanned for each new allocation,
+ * ie, 1/density of reusable buffers, and track a moving average of
+ * that.
+ *
+ * If the strategy point didn't move, we don't update the density
+ * estimate
+ */
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
+ {
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
+ }
- /*
- * Estimate how many reusable buffers there are between the current
- * strategy point and where we've scanned ahead to, based on the smoothed
- * density estimate.
- */
- bufs_ahead = NBuffers - bufs_to_lap;
- reusable_buffers_est = (float) bufs_ahead / smoothed_density;
+ /*
+ * Estimate how many reusable buffers there are between the current
+ * strategy point and where we've scanned ahead to, based on the
+ * smoothed density estimate.
+ */
+ bufs_ahead = num_buffers - bufs_to_lap;
+ reusable_buffers_est = (float) bufs_ahead / smoothed_density;
- /*
- * Track a moving average of recent buffer allocations. Here, rather than
- * a true average we want a fast-attack, slow-decline behavior: we
- * immediately follow any increase.
- */
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
- else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
- smoothing_samples;
+ /*
+ * Track a moving average of recent buffer allocations. Here, rather
+ * than a true average we want a fast-attack, slow-decline behavior:
+ * we immediately follow any increase.
+ */
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
+ else
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
+ smoothing_samples;
- /* Scale the estimate by a GUC to allow more aggressive tuning. */
- upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
+ /* Scale the estimate by a GUC to allow more aggressive tuning. */
+ upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
- /*
- * If recent_alloc remains at zero for many cycles, smoothed_alloc will
- * eventually underflow to zero, and the underflows produce annoying
- * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
- * zero, there's no point in tracking smaller and smaller values of
- * smoothed_alloc, so just reset it to exactly zero to avoid this
- * syndrome. It will pop back up as soon as recent_alloc increases.
- */
- if (upcoming_alloc_est == 0)
- smoothed_alloc = 0;
+ /*
+ * If recent_alloc remains at zero for many cycles, smoothed_alloc
+ * will eventually underflow to zero, and the underflows produce
+ * annoying kernel warnings on some platforms. Once
+ * upcoming_alloc_est has gone to zero, there's no point in tracking
+ * smaller and smaller values of smoothed_alloc, so just reset it to
+ * exactly zero to avoid this syndrome. It will pop back up as soon
+ * as recent_alloc increases.
+ */
+ if (upcoming_alloc_est == 0)
+ smoothed_alloc = 0;
- /*
- * Even in cases where there's been little or no buffer allocation
- * activity, we want to make a small amount of progress through the buffer
- * cache so that as many reusable buffers as possible are clean after an
- * idle period.
- *
- * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
- * the BGW will be called during the scan_whole_pool time; slice the
- * buffer pool into that many sections.
- */
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ /*
+ * Even in cases where there's been little or no buffer allocation
+ * activity, we want to make a small amount of progress through the
+ * buffer cache so that as many reusable buffers as possible are clean
+ * after an idle period.
+ *
+ * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many
+ * times the BGW will be called during the scan_whole_pool time; slice
+ * the buffer pool into that many sections.
+ */
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
- if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
- {
+ if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
+ {
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
- upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
+ elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
+ upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
#endif
- upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
- }
-
- /*
- * Now write out dirty reusable buffers, working forward from the
- * next_to_clean point, until we have lapped the strategy scan, or cleaned
- * enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.
- */
+ upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
+ }
- num_to_scan = bufs_to_lap;
- num_written = 0;
- reusable_buffers = reusable_buffers_est;
+ /*
+ * Now write out dirty reusable buffers, working forward from the
+ * next_to_clean point, until we have lapped the strategy scan, or
+ * cleaned enough buffers to match our estimate of the next cycle's
+ * allocation requirements, or hit the bgwriter_lru_maxpages limit.
+ */
- /* Execute the LRU scan */
- while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
- {
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ num_to_scan = bufs_to_lap;
+ num_written = 0;
+ reusable_buffers = reusable_buffers_est;
- if (++next_to_clean >= NBuffers)
+ /* Execute the LRU scan */
+ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- next_to_clean = 0;
- next_passes++;
- }
- num_to_scan--;
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
+ wb_context);
- if (sync_state & BUF_WRITTEN)
- {
- reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- PendingBgWriterStats.maxwritten_clean++;
- break;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
+ }
+ num_to_scan--;
+
+ if (sync_state & BUF_WRITTEN)
+ {
+ reusable_buffers++;
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
+ {
+ PendingBgWriterStats.maxwritten_clean++;
+ break;
+ }
}
+ else if (sync_state & BUF_REUSABLE)
+ reusable_buffers++;
}
- else if (sync_state & BUF_REUSABLE)
- reusable_buffers++;
- }
- PendingBgWriterStats.buf_written_clean += num_written;
+ PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
- elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
- smoothed_density, reusable_buffers_est, upcoming_alloc_est,
- bufs_to_lap - num_to_scan,
- num_written,
- reusable_buffers - reusable_buffers_est);
+ elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
+ smoothed_density, reusable_buffers_est, upcoming_alloc_est,
+ bufs_to_lap - num_to_scan,
+ num_written,
+ reusable_buffers - reusable_buffers_est);
#endif
- /*
- * Consider the above scan as being like a new allocation scan.
- * Characterize its density and update the smoothed one based on it. This
- * effectively halves the moving average period in cases where both the
- * strategy and the background writer are doing some useful scanning,
- * which is helpful because a long memory isn't as desirable on the
- * density estimates.
- */
- new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
- {
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
+ /*
+ * Consider the above scan as being like a new allocation scan.
+ * Characterize its density and update the smoothed one based on it.
+ * This effectively halves the moving average period in cases where
+ * both the strategy and the background writer are doing some useful
+ * scanning, which is helpful because a long memory isn't as desirable
+ * on the density estimates.
+ */
+ new_strategy_delta = bufs_to_lap - num_to_scan;
+ new_recent_alloc = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ {
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
- scans_per_alloc, smoothed_density);
+ elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
+ new_recent_alloc, new_strategy_delta,
+ scans_per_alloc, smoothed_density);
#endif
+ }
+
+ /* hibernate if all partitions can hibernate */
+ hibernate &= (bufs_to_lap == 0 && recent_alloc_partition == 0);
}
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index c3fbd651dd5..ff02dc8e00b 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -63,17 +63,27 @@ typedef struct BufferStrategyFreelist
#define MIN_FREELIST_PARTITIONS 4
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* Clock sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -83,6 +93,15 @@ typedef struct
*/
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
/*
* Bgworker process to be notified upon activity or -1 if none. See
@@ -99,6 +118,9 @@ typedef struct
int num_partitions;
int num_partitions_per_node;
+ /* clocksweep partitions */
+ ClockSweep *sweeps;
+
BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
@@ -138,6 +160,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -149,6 +172,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -156,14 +180,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -189,19 +213,23 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
}
static int
@@ -258,6 +286,28 @@ calculate_partition_index()
return index;
}
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
+}
+
/*
* ChooseFreeList
* Pick the buffer freelist to use, depending on the CPU and NUMA node.
@@ -374,7 +424,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -445,13 +495,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Nothing on the freelist, so run the "clock sweep" algorithm
*
- * XXX Should we also make this NUMA-aware, to only access buffers from
- * the same NUMA node? That'd probably mean we need to make the clock
- * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
- * subset of buffers. But that also means each process could "sweep" only
- * a fraction of buffers, even if the other buffers are better candidates
- * for eviction. Would that also mean we'd have multiple bgwriters, one
- * for each node, or would one bgwriter handle all of that?
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
*/
trycounter = NBuffers;
for (;;)
@@ -533,6 +587,41 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncStart -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ *num_buf_alloc += pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -540,37 +629,44 @@ StrategyFreeBuffer(BufferDesc *buf)
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -658,6 +754,10 @@ StrategyShmemSize(void)
size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
num_partitions)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -676,6 +776,7 @@ StrategyInitialize(bool init)
int num_nodes;
int num_partitions;
int num_partitions_per_node;
+ char *ptr;
/* */
BufferPartitionParams(&num_partitions, &num_nodes);
@@ -703,7 +804,8 @@ StrategyInitialize(bool init)
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
- MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -718,12 +820,41 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* have to point the sweeps array to right after the freelists */
+ ptr = (char *) StrategyControl +
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions);
+ StrategyControl->sweeps = (ClockSweep *) ptr;
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -771,7 +902,6 @@ StrategyInitialize(bool init)
buf->freeNext = freelist->firstFreeBuffer;
freelist->firstFreeBuffer = i;
}
-
}
}
else
@@ -1111,9 +1241,11 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
}
void
-FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
+ uint32 *complete_passes, uint32 *buffer_allocs, uint32 *next_victim_buffer)
{
BufferStrategyFreelist *freelist;
+ ClockSweep *sweep;
int cur;
/* stats */
@@ -1123,6 +1255,7 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
freelist = &StrategyControl->freelists[idx];
+ sweep = &StrategyControl->sweeps[idx];
/* stat */
SpinLockAcquire(&freelist->freelist_lock);
@@ -1152,4 +1285,11 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
*remain = cnt_remain;
*actually_free = cnt_free;
+
+ /* get the clocksweep stats too */
+ *complete_passes = sweep->completePasses;
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9dfbecb9fe4..907b160b4f7 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,7 +449,9 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index df127274190..53855d4be23 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -349,7 +349,10 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void FreelistPartitionGetInfo(int idx,
uint64 *consumed, uint64 *remain,
- uint64 *actually_free);
+ uint64 *actually_free,
+ uint32 *complete_passes,
+ uint32 *buffer_allocs,
+ uint32 *next_victim_buffer);
/* inline functions */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c695cfa76e8..a38dd8d6242 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.50.1
[text/x-patch] v3-0004-NUMA-partition-buffer-freelist.patch (20.9K, ../../[email protected]/5-v3-0004-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From 06a43d54498ab5049c12a458cfbd4fe3b3b168c2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v3 4/7] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
This uses the buffer partitioning scheme introduced by the earlier
patch, i.e. the partitions will "align" with NUMA nodes, etc.
It also extends the "pg_buffercache_partitions" view, to include
information about each freelist (number of consumedd buffers, ...).
When allocating a buffer, it's taken from the correct freelist (same
NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
.../pg_buffercache--1.6--1.7.sql | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/freelist.c | 360 ++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 4 +-
7 files changed, 372 insertions(+), 30 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index bd97246f6ab..3871c261528 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -12,7 +12,7 @@ LANGUAGE C PARALLEL SAFE;
-- Create a view for convenient access.
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
- (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer);
+ (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer, buffers_consumed bigint, buffers_remain bigint, buffers_free bigint);
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8baa7c7b543..668ada8c47b 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -812,6 +812,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "buffers_consumed",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "buffers_remain",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "buffers_free",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -833,12 +839,19 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffers_consumed,
+ buffers_remain,
+ buffers_free;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
+ &buffers_free);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -854,6 +867,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(buffers_consumed);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffers_remain);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffers_free);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..c3fbd651dd5 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,52 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
+
+/*
+ * The minimum number of partitions we want to have. We want at least this
+ * number of partitions, even on non-NUMA system, as it helps with contention
+ * for buffers. But with multiple NUMA nodes, we want a separate partition per
+ * node. But we may get multiple partitions per node, for low node count.
+ *
+ * With multiple partitions per NUMA node, we pick the partition based on CPU
+ * (or some other parameter).
+ */
+#define MIN_FREELIST_PARTITIONS 4
/*
* The shared freelist control information.
@@ -39,8 +77,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,8 +87,19 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -157,6 +204,88 @@ ClockSweepTick(void)
return victim;
}
+static int
+calculate_partition_index()
+{
+ int rc;
+ unsigned cpu;
+ unsigned node;
+ int index;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->freelists[index];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +297,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +325,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +392,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +442,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -356,7 +503,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ /*
+ * We don't want to call ChooseFreeList() again, because we might get a
+ * completely different freelist - either a different partition in the
+ * same group, or even a different group if the NUMA node changed. But we
+ * can calculate the proper freelist from the buffer id.
+ */
+ int index = (BufferGetNode(buf->buf_id) * StrategyControl->num_partitions_per_node)
+ + (buf->buf_id % StrategyControl->num_partitions_per_node);
+
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[index];
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +526,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +594,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int p = 0; p < StrategyControl->num_partitions; p++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[p];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "NUMA: freelist partition %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ p,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -445,12 +643,20 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+ int num_nodes;
+
+ BufferPartitionParams(&num_partitions, &num_nodes);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /* size of freelist partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ num_partitions)));
return size;
}
@@ -467,6 +673,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* */
+ BufferPartitionParams(&num_partitions, &num_nodes);
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -484,7 +702,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
&found);
if (!found)
@@ -494,13 +713,10 @@ StrategyInitialize(bool init)
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +727,52 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < num_partitions; nfreelist++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(nfreelist, &node,
+ &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * Walk through buffers for each partition, add them to the list.
+ * Walk from the end, because we're adding the buffers to the
+ * beginning.
+ */
+
+ for (int i = last_buffer; i >= first_buffer; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+
+ /* add to the freelist */
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
+
+ }
}
else
Assert(!init);
@@ -847,3 +1109,47 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+{
+ BufferStrategyFreelist *freelist;
+ int cur;
+
+ /* stats */
+ uint64 cnt_remain = 0;
+ uint64 cnt_free = 0;
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[idx];
+
+ /* stat */
+ SpinLockAcquire(&freelist->freelist_lock);
+
+ *consumed = freelist->consumed;
+
+ cur = freelist->firstFreeBuffer;
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ cnt_remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ cnt_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ SpinLockRelease(&freelist->freelist_lock);
+
+ *remain = cnt_remain;
+ *actually_free = cnt_free;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..a11bc71a386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+bool numa_partition_freelist = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a21f20800fb..0552ed62cc7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2136,6 +2136,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..66baf2bf33e 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index deaf4f19fa4..df127274190 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -347,7 +347,9 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
-
+extern void FreelistPartitionGetInfo(int idx,
+ uint64 *consumed, uint64 *remain,
+ uint64 *actually_free);
/* inline functions */
--
2.50.1
[text/x-patch] v3-0003-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../../[email protected]/6-v3-0003-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From 3c7dbbec4ee3957c92bf647605768beb5473f66b Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v3 3/7] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.50.1
[text/x-patch] v3-0002-NUMA-localalloc.patch (3.7K, ../../[email protected]/7-v3-0002-NUMA-localalloc.patch)
download | inline diff:
From d08a94b0f54a4c4986d351f98269905bb511624c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v3 2/7] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 17 +++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 29 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 43b4dbccc3d..079974944e9 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,19 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9570087aa60..a21f20800fb 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v3-0001-NUMA-interleaving-buffers.patch (38.2K, ../../[email protected]/8-v3-0001-NUMA-interleaving-buffers.patch)
download | inline diff:
From f0eb1af6fdcfd7daae26952ddc223952333f6af2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 28 Jul 2025 14:01:37 +0200
Subject: [PATCH v3 1/7] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to 'off'), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by numa_buffers_interleave GUC (default: false)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
contrib/pg_buffercache/Makefile | 2 +-
.../pg_buffercache--1.6--1.7.sql | 22 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 626 +++++++++++++++++-
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 2 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 15 +
src/tools/pgindent/typedefs.list | 2 +
11 files changed, 771 insertions(+), 11 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..bd97246f6ab
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,22 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae0291e6e96..8baa7c7b543 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -771,3 +773,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..5b65a855b29 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +32,19 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsArray;
+
+static Size get_memory_page_size(void);
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
@@ -70,19 +91,89 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +203,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize the registry of buffer partitions, and also move the
+ * memory to different NUMA nodes (if enabled by GUC)
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +241,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +254,68 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -184,5 +330,467 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+
+#define MIN_BUFFER_PARTITIONS 4
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(LOG, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (numa_partitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ part->numa_node = n;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(LOG, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (!numa_buffers_interleave || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ *num_partitions = numa_partitions;
+ *num_nodes = numa_nodes;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9570087aa60 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 52a71b138f7..9dfbecb9fe4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -323,6 +323,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -491,4 +492,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..deaf4f19fa4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -143,6 +143,20 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+typedef struct BufferPartition
+{
+ int numa_node;
+ int num_buffers;
+ int first_buffer;
+ int last_buffer;
+} BufferPartition;
+
+typedef struct BufferPartitions
+{
+ int npartitions;
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* forward declared, to avoid having to expose buf_internals.h here */
struct WritebackContext;
@@ -319,6 +333,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3daba26b237..c695cfa76e8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.50.1
[text/csv] numa-hb176.csv (17.8K, ../../[email protected]/9-numa-hb176.csv)
download | inline:
shared_buffers mode build pinning tps tps_25 tps_50 tps_75 tps_95 tps_99 lat_avg lat_stddev sb_warmup sb_benchmark
32GB simple numa-0-master none 2094014.794767 2073550.4 2082788.6 2102633.4 2167920.9 2184485.4 0.061 0.020 100.00 100.00
32GB simple numa-0-master random 1286271.933424 1284754.0 1286494.7 1288674.1 1291197.6 1292495.7 0.099 0.036 100.00 100.00
32GB simple numa-0-master colocated 2388811.232587 2383281.7 2389972.1 2396598.3 2401910.3 2404253.9 0.053 0.012 100.00 100.00
32GB simple numa-1-buffers none 2047019.576932 2033460.0 2042257.4 2055448.4 2085925.9 2114769.4 0.062 0.024 100.00 100.00
32GB simple numa-1-buffers random 1192511.104640 1191714.2 1192982.1 1194393.7 1196299.0 1197481.5 0.107 0.035 100.00 100.00
32GB simple numa-1-buffers colocated 2405932.896675 2398925.4 2407886.6 2415869.6 2420222.4 2422280.8 0.053 0.012 100.00 100.00
32GB simple numa-2-localalloc none 2069848.559853 2055515.7 2066642.7 2079182.0 2115517.3 2130680.9 0.062 0.028 100.00 100.00
32GB simple numa-2-localalloc random 1206669.031819 1205724.8 1207054.5 1208558.7 1210337.3 1211225.3 0.106 0.034 100.00 100.00
32GB simple numa-2-localalloc colocated 2357590.782369 2354531.3 2359009.7 2363536.6 2367643.1 2368925.9 0.054 0.012 100.00 100.00
32GB simple numa-3-no-tail none 2076592.800331 2056606.8 2069066.2 2087446.7 2140462.9 2159359.8 0.061 0.025 100.00 100.00
32GB simple numa-3-no-tail random 1201742.073521 1223346.1 1224697.3 1226182.0 1227864.0 1228923.4 0.106 0.038 100.00 100.00
32GB simple numa-3-no-tail colocated 2404952.281461 2397528.5 2406571.7 2414327.2 2419760.7 2422838.3 0.053 0.012 100.00 100.00
32GB simple numa-4-freelist none 2144928.827252 2127996.5 2140287.4 2156668.7 2200868.4 2215117.6 0.060 0.049 100.00 100.00
32GB simple numa-4-freelist random 1262573.059680 1262244.5 1263434.2 1264747.4 1266127.9 1267152.6 0.101 0.054 100.00 100.00
32GB simple numa-4-freelist colocated 2452015.746479 2445803.4 2453721.3 2461334.6 2465615.6 2466924.9 0.052 0.042 100.00 100.00
32GB simple numa-5-clocksweep none 2147727.945415 2130889.9 2146203.3 2162137.9 2189644.5 2227659.6 0.059 0.035 25.00 100.00
32GB simple numa-5-clocksweep random 1241338.787551 1241416.4 1242391.2 1243166.6 1243964.3 1244688.1 0.103 0.045 25.00 100.00
32GB simple numa-5-clocksweep colocated 2545190.769146 2545681.2 2546922.8 2547753.0 2548595.1 2549195.3 0.050 0.028 25.00 100.00
32GB simple numa-6-pgproc none 2180410.964299 2157924.5 2170931.6 2192210.9 2249331.0 2272390.9 0.059 0.045 25.00 100.00
32GB simple numa-6-pgproc random 1240799.452682 1240842.3 1241607.5 1242474.2 1243488.1 1243901.1 0.103 0.045 25.00 100.00
32GB simple numa-6-pgproc colocated 2636934.271698 2637377.9 2638581.9 2639457.1 2641224.1 2642282.5 0.048 0.023 25.00 100.00
32GB simple numa-7-pinning none 2101452.917588 2090545.5 2103858.4 2118982.5 2130923.9 2135597.8 0.061 0.030 25.00 100.00
32GB simple numa-7-pinning random 1238919.511046 1238852.6 1239708.4 1240412.1 1241329.5 1241757.2 0.103 0.046 25.00 100.00
32GB simple numa-7-pinning colocated 2612797.027980 2613302.4 2614308.2 2615035.3 2616057.0 2616497.2 0.049 0.031 25.00 100.00
32GB prepared numa-0-master none 2600265.035719 2594175.1 2603657.8 2614298.2 2627655.9 2639079.8 0.049 0.032 100.00 100.00
32GB prepared numa-0-master random 609278.573839 601973.9 608734.8 615687.7 626392.1 643183.4 0.209 0.082 100.00 100.00
32GB prepared numa-0-master colocated 2936188.002438 2931071.7 2937149.1 2942793.5 2951091.2 2955221.7 0.043 0.015 100.00 100.00
32GB prepared numa-1-buffers none 2556483.026566 2547839.7 2558098.4 2568563.8 2582199.4 2597160.4 0.050 0.034 100.00 100.00
32GB prepared numa-1-buffers random 643264.098458 638733.7 645325.8 651338.1 657511.7 659828.3 0.198 0.086 100.00 100.00
32GB prepared numa-1-buffers colocated 2896245.824802 2885755.9 2896842.2 2906640.6 2918054.9 2922427.6 0.044 0.014 100.00 100.00
32GB prepared numa-2-localalloc none 2595430.169568 2584462.8 2597788.8 2609027.1 2623358.2 2635969.4 0.049 0.034 100.00 100.00
32GB prepared numa-2-localalloc random 594695.123377 589296.0 593619.3 600417.6 609539.2 612091.0 0.214 0.081 100.00 100.00
32GB prepared numa-2-localalloc colocated 2918297.926587 2909171.3 2918484.3 2927639.3 2938536.1 2943213.7 0.044 0.014 100.00 100.00
32GB prepared numa-3-no-tail none 2572010.525302 2561524.7 2574921.5 2586000.7 2604077.8 2617103.1 0.050 0.034 100.00 100.00
32GB prepared numa-3-no-tail random 607225.735077 598681.9 604986.4 615368.3 627310.5 634224.9 0.210 0.080 100.00 100.00
32GB prepared numa-3-no-tail colocated 2903377.427631 2899103.0 2904451.2 2908960.5 2914982.7 2919139.5 0.044 0.014 100.00 100.00
32GB prepared numa-4-freelist none 2656080.200019 2647042.2 2657972.3 2671358.7 2694405.7 2707741.6 0.048 0.042 100.00 100.00
32GB prepared numa-4-freelist random 1487277.158053 1492190.7 1493668.9 1495124.8 1496979.6 1498150.4 0.086 0.041 100.00 100.00
32GB prepared numa-4-freelist colocated 3033967.176021 3029920.1 3034382.4 3038608.2 3045720.7 3047922.2 0.042 0.022 100.00 100.00
32GB prepared numa-5-clocksweep none 2832317.609128 2823434.5 2836798.6 2851170.0 2872311.7 2885272.1 0.045 0.041 25.00 100.00
32GB prepared numa-5-clocksweep random 1461263.713983 1480257.7 1481681.0 1482839.0 1484390.0 1485436.7 0.087 0.044 25.00 100.00
32GB prepared numa-5-clocksweep colocated 3287319.038876 3287489.6 3288874.0 3289849.3 3291265.6 3291769.0 0.039 0.024 25.00 100.00
32GB prepared numa-6-pgproc none 2864731.663543 2856449.9 2868483.2 2880807.6 2897585.8 2912240.6 0.045 0.040 25.00 100.00
32GB prepared numa-6-pgproc random 1490324.471864 1507399.4 1509042.0 1510167.1 1511607.1 1512726.9 0.086 0.044 25.00 100.00
32GB prepared numa-6-pgproc colocated 3332521.391978 3333023.7 3333983.0 3334822.0 3336166.3 3336745.6 0.038 0.028 25.00 100.00
32GB prepared numa-7-pinning none 3196743.972868 3195967.9 3204768.7 3214707.2 3228582.4 3235446.2 0.040 0.032 25.00 100.00
32GB prepared numa-7-pinning random 1497592.952405 1515042.8 1516513.1 1517717.4 1519105.0 1519837.0 0.085 0.043 25.00 100.00
32GB prepared numa-7-pinning colocated 3366788.066030 3364820.9 3366270.6 3371283.5 3374229.5 3375696.5 0.038 0.023 25.00 100.00
16GB simple numa-0-master none 1827078.467315 1817756.3 1825443.1 1832543.8 1859686.9 1870789.3 0.070 0.024 100.00 100.00
16GB simple numa-0-master random 1204011.746569 1203116.3 1204501.7 1205861.8 1207978.8 1208888.9 0.106 0.051 100.00 100.00
16GB simple numa-0-master colocated 2091429.835432 2091732.8 2092492.3 2093089.4 2094002.7 2094487.2 0.061 0.016 100.00 100.00
16GB simple numa-1-buffers none 1818607.320519 1810580.9 1817367.6 1825469.1 1841348.0 1852768.7 0.070 0.029 100.00 100.00
16GB simple numa-1-buffers random 1137821.791547 1136829.3 1138436.1 1139733.2 1141115.9 1142075.5 0.112 0.043 100.00 100.00
16GB simple numa-1-buffers colocated 2070456.921423 2070408.5 2071031.2 2071629.6 2072642.0 2073077.9 0.062 0.015 100.00 100.00
16GB simple numa-2-localalloc none 1814337.964368 1802150.1 1809157.6 1818835.4 1859212.0 1879167.6 0.070 0.030 100.00 100.00
16GB simple numa-2-localalloc random 1140136.209840 1139137.5 1140471.3 1141764.2 1143336.5 1144688.4 0.112 0.045 100.00 100.00
16GB simple numa-2-localalloc colocated 2046749.245045 2046747.3 2047343.3 2047936.5 2048622.9 2049092.8 0.062 0.015 100.00 100.00
16GB simple numa-3-no-tail none 1788677.591511 1779668.7 1786752.4 1794861.3 1816346.7 1834453.2 0.071 0.037 100.00 100.00
16GB simple numa-3-no-tail random 1147815.761542 1146738.5 1148289.5 1149497.5 1151206.6 1152309.6 0.111 0.044 100.00 100.00
16GB simple numa-3-no-tail colocated 2063855.159562 2063587.2 2064405.6 2065153.3 2065818.4 2066994.5 0.062 0.015 100.00 100.00
16GB simple numa-4-freelist none 1872879.183670 1861914.8 1869084.4 1878799.0 1913317.6 1933159.1 0.068 0.033 100.00 100.00
16GB simple numa-4-freelist random 1207729.009946 1206622.2 1208305.2 1209726.6 1211618.5 1213711.5 0.106 0.053 100.00 100.00
16GB simple numa-4-freelist colocated 2135628.758077 2135447.6 2136058.9 2136774.1 2137744.9 2138242.8 0.060 0.026 100.00 100.00
16GB simple numa-5-clocksweep none 2016983.483955 2002764.2 2014062.2 2026487.8 2060396.1 2069378.2 0.063 0.034 25.00 100.00
16GB simple numa-5-clocksweep random 1215517.698425 1214417.6 1215885.8 1217436.2 1219034.0 1220391.1 0.105 0.042 25.00 100.00
16GB simple numa-5-clocksweep colocated 2365377.989509 2365342.3 2365867.5 2366356.9 2367174.3 2367813.4 0.054 0.015 25.00 100.00
16GB simple numa-6-pgproc none 1988517.293447 1976851.9 1985550.0 1997500.1 2022116.5 2040373.8 0.064 0.038 25.00 100.00
16GB simple numa-6-pgproc random 1179440.500877 1178373.3 1179920.9 1181132.7 1182906.5 1184106.3 0.108 0.048 25.01 100.00
16GB simple numa-6-pgproc colocated 2345242.334786 2345206.5 2345775.2 2346357.8 2347204.3 2347987.4 0.054 0.017 25.01 100.00
16GB simple numa-7-pinning none 2143131.371367 2137376.4 2145502.7 2153375.1 2160641.8 2163537.3 0.060 0.028 25.01 100.00
16GB simple numa-7-pinning random 1188670.161623 1187596.6 1189122.9 1190417.1 1192084.0 1193320.3 0.108 0.045 25.00 100.00
16GB simple numa-7-pinning colocated 2403120.900906 2402640.4 2403340.4 2404187.0 2406320.0 2407336.8 0.053 0.016 25.01 100.00
16GB prepared numa-0-master none 2221009.728590 2217463.4 2226291.2 2234013.4 2245377.8 2256120.3 0.057 0.041 100.00 100.00
16GB prepared numa-0-master random 602510.378457 593188.6 603924.9 610132.7 619242.5 633291.3 0.212 0.091 100.00 100.00
16GB prepared numa-0-master colocated 2481978.582055 2481482.9 2482651.3 2483698.2 2484969.3 2485873.2 0.051 0.018 100.00 100.00
16GB prepared numa-1-buffers none 2193555.966068 2189913.6 2198171.8 2207163.1 2218853.3 2229229.3 0.058 0.043 100.00 100.00
16GB prepared numa-1-buffers random 613141.518268 603012.6 611405.3 622477.1 633218.1 648076.5 0.208 0.078 100.00 100.00
16GB prepared numa-1-buffers colocated 2496646.168791 2496647.0 2497740.0 2498703.0 2499768.5 2500156.5 0.051 0.017 100.00 100.00
16GB prepared numa-2-localalloc none 2213277.504474 2208962.0 2218383.2 2227701.0 2239536.7 2251144.5 0.058 0.043 100.00 100.00
16GB prepared numa-2-localalloc random 618355.985374 611937.5 618198.9 623245.7 636857.7 644933.8 0.207 0.081 100.00 100.00
16GB prepared numa-2-localalloc colocated 2502981.656459 2502711.5 2503524.0 2504358.4 2505381.9 2506685.0 0.051 0.017 100.00 100.00
16GB prepared numa-3-no-tail none 2211849.394014 2206713.7 2216419.3 2225485.4 2237445.0 2248063.5 0.058 0.043 100.00 100.00
16GB prepared numa-3-no-tail random 575000.639904 570502.7 576741.1 580100.8 585510.7 590986.2 0.222 0.096 100.00 100.00
16GB prepared numa-3-no-tail colocated 2460474.838856 2460029.2 2461049.4 2462003.9 2463196.9 2463833.2 0.052 0.018 100.00 100.00
16GB prepared numa-4-freelist none 2281726.084202 2277193.2 2288404.0 2298464.9 2310718.6 2319249.1 0.056 0.045 100.00 100.00
16GB prepared numa-4-freelist random 1398663.617786 1398797.0 1400908.7 1403105.2 1406152.4 1409074.7 0.091 0.052 100.00 100.00
16GB prepared numa-4-freelist colocated 2679995.292637 2679985.9 2680862.0 2681547.4 2682665.8 2683429.4 0.048 0.028 100.00 100.00
16GB prepared numa-5-clocksweep none 2540095.692745 2532340.2 2545940.3 2560762.3 2577130.8 2589321.1 0.050 0.046 25.01 100.00
16GB prepared numa-5-clocksweep random 1453707.189064 1459045.1 1461321.8 1463586.2 1466581.8 1467714.9 0.088 0.050 25.00 100.00
16GB prepared numa-5-clocksweep colocated 2868908.990574 2868459.3 2869305.9 2869931.4 2870962.4 2872039.0 0.045 0.019 25.01 100.00
16GB prepared numa-6-pgproc none 2514531.754697 2508478.5 2520232.0 2532724.4 2555334.2 2572128.4 0.051 0.051 25.00 100.00
16GB prepared numa-6-pgproc random 1421942.962229 1425471.6 1427251.5 1429185.0 1432119.4 1434213.6 0.090 0.048 25.01 100.00
16GB prepared numa-6-pgproc colocated 2895460.904497 2894977.4 2895783.9 2896473.0 2897331.0 2897774.0 0.044 0.020 25.01 100.00
16GB prepared numa-7-pinning none 2570293.162052 2566575.5 2577298.2 2586033.6 2597683.7 2605701.0 0.050 0.045 25.01 100.00
16GB prepared numa-7-pinning random 1419469.356241 1422140.4 1424287.1 1425895.0 1429015.0 1431937.3 0.090 0.048 25.00 100.00
16GB prepared numa-7-pinning colocated 2965399.304227 2964597.4 2965842.7 2966982.3 2968241.0 2969082.9 0.043 0.018 25.00 100.00
48GB simple numa-0-master none 2240626.513398 2217063.6 2239195.3 2265723.7 2301549.5 2341012.7 0.057 0.118 83.07 100.00
48GB simple numa-0-master random 1318468.781539 1319007.2 1320126.3 1321273.5 1322828.4 1323737.2 0.097 0.162 84.09 100.00
48GB simple numa-0-master colocated 2699394.144782 2688747.8 2697168.5 2716980.8 2731581.3 2733854.0 0.047 0.108 82.93 100.00
48GB simple numa-1-buffers none 2245154.802833 2226344.9 2240563.6 2260432.1 2315478.3 2349275.6 0.057 0.122 83.31 100.00
48GB simple numa-1-buffers random 1261172.614367 1262162.4 1262880.0 1263940.7 1265028.8 1265850.8 0.101 0.164 83.59 100.00
48GB simple numa-1-buffers colocated 2650623.262153 2642051.6 2648415.7 2668555.3 2680209.3 2683861.3 0.048 0.108 81.91 100.00
48GB simple numa-2-localalloc none 2249367.782141 2228686.8 2248606.8 2273746.4 2312259.7 2331650.3 0.057 0.136 82.87 100.00
48GB simple numa-2-localalloc random 1244674.409409 1245027.6 1246358.7 1247559.9 1248836.7 1249097.2 0.103 0.164 82.72 100.00
48GB simple numa-2-localalloc colocated 2611193.021413 2600647.0 2609337.8 2628624.3 2640684.6 2643435.3 0.049 0.109 82.64 100.00
48GB simple numa-3-no-tail none 2268008.571187 2240616.6 2261884.9 2296781.3 2348528.0 2368243.7 0.056 0.119 82.66 100.00
48GB simple numa-3-no-tail random 1233026.884272 1233540.6 1234897.1 1236002.0 1237262.7 1237872.8 0.104 0.159 82.54 100.00
48GB simple numa-3-no-tail colocated 2658156.684861 2651293.0 2656574.6 2673886.2 2682779.0 2684405.4 0.048 0.114 82.15 100.00
48GB simple numa-4-freelist none 2251576.603050 2230542.2 2244920.3 2270473.5 2337061.4 2352738.5 0.057 0.150 83.02 100.00
48GB simple numa-4-freelist random 1307933.407227 1308092.8 1309804.3 1311297.6 1312511.3 1313296.8 0.098 0.135 82.57 100.00
48GB simple numa-4-freelist colocated 2601537.595536 2584555.7 2598362.4 2622709.5 2638031.8 2640277.4 0.049 0.072 81.97 100.00
48GB simple numa-5-clocksweep none 2241382.241155 2218548.3 2237248.1 2265353.8 2295880.7 2323149.0 0.057 0.038 25.00 100.00
48GB simple numa-5-clocksweep random 1262514.317310 1263358.9 1264072.0 1264592.4 1265358.6 1265534.2 0.101 0.053 25.00 100.00
48GB simple numa-5-clocksweep colocated 2610661.390468 2611730.6 2612727.7 2613709.4 2614884.8 2615970.1 0.049 0.030 25.00 100.00
48GB simple numa-6-pgproc none 2336305.373280 2301276.0 2324107.3 2360639.4 2436318.1 2466602.9 0.055 0.037 25.00 100.00
48GB simple numa-6-pgproc random 1273021.136908 1273754.9 1274463.0 1275048.7 1275733.1 1275980.7 0.100 0.054 25.00 100.00
48GB simple numa-6-pgproc colocated 2774107.243898 2775455.7 2776917.8 2777976.2 2779298.2 2780131.0 0.046 0.032 25.00 100.00
48GB simple numa-7-pinning none 2415318.045028 2407691.0 2422663.2 2433578.9 2450156.5 2467841.0 0.053 0.043 25.00 100.00
48GB simple numa-7-pinning random 1252468.481863 1253129.0 1253795.5 1254362.7 1254986.2 1255774.8 0.102 0.054 25.00 100.00
48GB simple numa-7-pinning colocated 2780002.945371 2781441.5 2783055.8 2784023.5 2785046.9 2785595.3 0.046 0.029 25.00 100.00
48GB prepared numa-0-master none 2923496.551126 2905774.4 2922465.4 2939254.6 2963572.9 2984084.0 0.044 0.029 100.00 100.00
48GB prepared numa-0-master random 587603.235381 582356.7 586515.2 591790.0 598064.6 627649.2 0.218 0.087 100.00 100.00
48GB prepared numa-0-master colocated 3299794.118436 3287550.0 3294055.9 3306638.1 3332989.3 3335413.0 0.039 0.011 100.00 100.00
48GB prepared numa-1-buffers none 2831238.524941 2814327.5 2830103.8 2847890.1 2873928.6 2888011.2 0.045 0.030 100.00 100.00
48GB prepared numa-1-buffers random 609794.507428 602144.0 607516.3 618335.2 626322.2 632261.4 0.210 0.084 100.00 100.00
48GB prepared numa-1-buffers colocated 3231774.005677 3218329.5 3225551.4 3244716.2 3263817.8 3269024.8 0.039 0.011 100.00 100.00
48GB prepared numa-2-localalloc none 2890615.623142 2871459.0 2889893.6 2906771.8 2928779.7 2949137.3 0.044 0.029 100.00 100.00
48GB prepared numa-2-localalloc random 597263.592382 589548.2 595436.5 602463.6 614818.7 624720.2 0.214 0.085 100.00 100.00
48GB prepared numa-2-localalloc colocated 3313277.527719 3298239.7 3307092.5 3327513.0 3345337.3 3351738.9 0.039 0.011 100.00 100.00
48GB prepared numa-3-no-tail none 2880257.694598 2860167.2 2877982.3 2895775.0 2925540.9 2942045.3 0.044 0.028 100.00 100.00
48GB prepared numa-3-no-tail random 598932.466701 593835.6 599657.4 604247.1 610151.9 613496.6 0.213 0.084 100.00 100.00
48GB prepared numa-3-no-tail colocated 3242629.469947 3230052.0 3236094.3 3254957.5 3274484.5 3279944.4 0.039 0.011 100.00 100.00
48GB prepared numa-4-freelist none 2923127.852195 2908968.9 2926129.9 2947127.7 2979743.3 2997703.0 0.044 0.157 100.00 100.00
48GB prepared numa-4-freelist random 1513129.413478 1515095.4 1516737.2 1518212.7 1519615.5 1520348.8 0.084 0.152 100.00 100.00
48GB prepared numa-4-freelist colocated 3395256.901017 3387587.5 3390973.7 3406014.7 3425695.7 3431188.9 0.038 0.070 100.00 100.00
48GB prepared numa-5-clocksweep none 2965437.145038 2949924.3 2965560.5 2985599.7 3008788.0 3020367.0 0.043 0.041 25.00 100.00
48GB prepared numa-5-clocksweep random 632449.314565 603792.6 613986.6 638880.6 758179.3 824435.2 0.202 0.099 25.00 100.00
48GB prepared numa-5-clocksweep colocated 3476865.561664 3477458.2 3478930.4 3480293.6 3482151.3 3483126.4 0.037 0.028 25.00 100.00
48GB prepared numa-6-pgproc none 3005569.927865 2989976.9 3007566.8 3023434.9 3052049.5 3073108.0 0.042 0.041 25.00 100.00
48GB prepared numa-6-pgproc random 607680.562253 579408.9 584803.8 600815.2 689898.4 975531.4 0.210 0.104 25.00 100.00
48GB prepared numa-6-pgproc colocated 3486467.905993 3487855.4 3489198.7 3490324.1 3492199.3 3492894.6 0.037 0.035 25.00 100.00
48GB prepared numa-7-pinning none 3177862.124437 3171786.2 3185876.7 3201378.0 3214490.6 3216137.0 0.040 0.045 25.00 100.00
48GB prepared numa-7-pinning random 626214.083568 602386.2 610697.5 619334.3 742241.2 906615.1 0.204 0.095 25.00 100.00
48GB prepared numa-7-pinning colocated 3562390.486753 3563016.5 3564196.8 3565135.0 3566639.3 3568153.2 0.036 0.029 25.00 100.00
[application/x-shellscript] run-huge-pages.sh (4.6K, ../../[email protected]/10-run-huge-pages.sh)
download
[text/csv] numa-xeon.csv (17.0K, ../../[email protected]/11-numa-xeon.csv)
download | inline:
shared_buffers mode build tps tps_25 tps_50 tps_75 tps_95 tps_99 lat_avg lat_stddev sb_warmup sb_benchmark
16GB simple numa-0-master none 435660.835849 436569.7 437207.0 437769.1 438452.7 439080.9 0.101 0.011 100.00 100.00
16GB simple numa-0-master random 435079.457369 437119.3 437506.1 437756.3 438134.0 438291.5 0.101 0.024 100.00 100.00
16GB simple numa-0-master colocated 436358.722452 437406.6 437615.3 437874.0 438046.3 438115.3 0.100 0.013 100.00 100.00
16GB simple numa-1-buffers none 439427.372422 440365.9 440997.7 441622.7 442693.4 443225.0 0.100 0.010 100.00 100.00
16GB simple numa-1-buffers random 435027.840470 437371.8 437548.6 437674.4 437835.1 437977.1 0.101 0.024 100.00 100.00
16GB simple numa-1-buffers colocated 450595.291038 451412.6 451896.2 452269.9 452706.8 452926.4 0.097 0.014 100.00 100.00
16GB simple numa-2-localalloc none 436134.188747 437118.8 437674.4 438109.5 439128.5 439655.1 0.101 0.011 100.00 100.00
16GB simple numa-2-localalloc random 437572.460095 439665.5 440029.2 440257.7 440517.3 440679.6 0.100 0.023 100.00 100.00
16GB simple numa-2-localalloc colocated 444300.471660 445406.9 445618.0 445825.0 446094.6 446246.4 0.099 0.014 100.00 100.00
16GB simple numa-3-no-tail none 442966.018997 443868.4 444521.3 445183.2 446048.3 446656.7 0.099 0.023 100.00 100.00
16GB simple numa-3-no-tail random 424481.739349 426572.7 426769.8 426986.1 427158.8 427255.8 0.103 0.024 100.00 100.00
16GB simple numa-3-no-tail colocated 446510.670808 447743.0 447996.2 448224.2 448483.6 448665.8 0.098 0.014 100.00 100.00
16GB simple numa-4-freelist none 434221.725822 435234.3 435867.1 436456.9 437180.8 437682.7 0.101 0.024 100.00 100.00
16GB simple numa-4-freelist random 433152.504959 435534.9 435810.3 436072.8 436416.5 436578.0 0.101 0.028 100.00 100.00
16GB simple numa-4-freelist colocated 455622.595281 456832.0 457102.8 457341.8 457665.1 457882.3 0.096 0.019 100.00 100.00
16GB simple numa-5-clocksweep none 440814.327529 441999.8 442477.1 443091.1 443836.2 444687.0 0.099 0.015 50.00 100.00
16GB simple numa-5-clocksweep random 432098.974783 434509.3 434730.0 434871.7 435060.8 435180.0 0.102 0.027 25.01 100.00
16GB simple numa-5-clocksweep colocated 447123.967715 448425.4 448583.4 448774.8 448957.4 449042.6 0.098 0.018 50.00 100.00
16GB simple numa-6-pgproc none 444542.151597 445826.3 446356.1 446856.9 447681.5 448287.1 0.099 0.017 25.01 100.00
16GB simple numa-6-pgproc random 437828.906615 440373.6 440689.5 440846.6 441058.2 441178.0 0.100 0.028 25.01 100.00
16GB simple numa-6-pgproc colocated 455042.538054 456570.6 456712.2 456849.8 457011.3 457107.5 0.096 0.017 50.01 100.00
16GB simple numa-7-pinning none 441672.926416 442804.5 443432.7 444105.0 445128.0 446039.1 0.099 0.017 26.86 100.00
16GB simple numa-7-pinning random 426223.045224 428477.9 428677.4 428851.0 429036.2 429105.0 0.103 0.026 50.01 100.00
16GB simple numa-7-pinning colocated 450392.113450 451822.6 451924.7 452028.9 452180.4 452238.1 0.097 0.017 50.01 100.00
16GB prepared numa-0-master none 780270.504321 781452.2 783365.3 784805.8 787342.7 788523.2 0.056 0.008 100.00 100.00
16GB prepared numa-0-master random 740319.460261 744682.7 745343.8 745970.8 746679.9 747127.0 0.059 0.022 100.00 100.00
16GB prepared numa-0-master colocated 808745.174879 810974.9 811570.6 812054.2 812866.5 813370.0 0.054 0.010 100.00 100.00
16GB prepared numa-1-buffers none 779061.549666 780355.0 782098.7 783859.4 786243.6 787488.0 0.056 0.008 100.00 100.00
16GB prepared numa-1-buffers random 731931.525311 735979.8 736774.3 737560.1 738127.5 738339.2 0.060 0.022 100.00 100.00
16GB prepared numa-1-buffers colocated 834043.848216 835960.7 836704.2 837765.5 838531.0 838780.4 0.052 0.010 100.00 100.00
16GB prepared numa-2-localalloc none 776686.423107 778070.3 779738.9 781332.7 783734.7 784749.5 0.056 0.008 100.00 100.00
16GB prepared numa-2-localalloc random 746071.021032 750606.0 751054.7 751520.2 752190.4 752477.2 0.059 0.022 100.00 100.00
16GB prepared numa-2-localalloc colocated 845194.301526 847919.8 848270.9 848585.7 849056.6 849324.5 0.052 0.011 100.00 100.00
16GB prepared numa-3-no-tail none 767472.661364 768654.5 770426.5 772322.9 774160.3 775354.7 0.057 0.008 100.00 100.00
16GB prepared numa-3-no-tail random 735100.627273 739765.5 740399.9 740837.1 741499.4 741808.5 0.060 0.022 100.00 100.00
16GB prepared numa-3-no-tail colocated 824897.098984 827288.2 827852.2 828430.8 828920.2 829226.3 0.053 0.010 100.00 100.00
16GB prepared numa-4-freelist none 759776.605571 760861.3 762909.4 764281.0 766359.8 768521.9 0.058 0.014 100.00 100.00
16GB prepared numa-4-freelist random 729800.689065 734337.4 734947.1 735428.8 736218.4 736975.4 0.060 0.041 100.00 100.00
16GB prepared numa-4-freelist colocated 846908.222612 849617.2 850197.0 850741.3 851328.2 851774.7 0.052 0.019 100.00 100.00
16GB prepared numa-5-clocksweep none 783444.253474 785122.5 786840.3 788354.2 790602.0 792040.7 0.056 0.012 50.01 100.00
16GB prepared numa-5-clocksweep random 738274.074795 742770.6 743492.0 743935.3 744471.7 744799.4 0.059 0.023 50.00 100.00
16GB prepared numa-5-clocksweep colocated 874129.726758 877734.8 878238.5 878509.2 878937.3 879362.2 0.050 0.013 50.01 100.00
16GB prepared numa-6-pgproc none 776320.601272 777672.8 779517.1 781352.5 783360.7 786641.0 0.056 0.012 50.00 100.00
16GB prepared numa-6-pgproc random 748473.862569 752733.0 753614.7 754762.8 755471.1 755954.3 0.059 0.024 50.00 100.00
16GB prepared numa-6-pgproc colocated 858900.054256 861867.8 862316.6 862654.1 863077.5 863358.5 0.051 0.013 50.01 100.00
16GB prepared numa-7-pinning none 778286.188920 780018.7 781462.8 782675.8 784586.6 786637.1 0.056 0.014 25.01 100.00
16GB prepared numa-7-pinning random 746250.393130 751720.5 752065.0 752399.7 752909.5 753307.0 0.059 0.032 25.01 100.00
16GB prepared numa-7-pinning colocated 860645.977827 863759.1 864312.1 864970.8 865869.3 866131.5 0.051 0.014 25.01 100.00
8GB simple numa-0-master none 420481.702806 421397.6 421991.9 422476.7 423331.7 423793.6 0.104 0.012 100.00 100.00
8GB simple numa-0-master random 406979.498757 408972.2 409116.6 409268.8 409457.2 409527.0 0.108 0.026 100.00 100.00
8GB simple numa-0-master colocated 421026.362476 421910.1 422224.4 422695.3 422846.0 422907.7 0.104 0.016 100.00 100.00
8GB simple numa-1-buffers none 420127.755712 421044.4 421598.3 422144.6 423093.7 423777.7 0.104 0.012 100.00 100.00
8GB simple numa-1-buffers random 408125.575066 410247.1 410399.4 410546.2 410731.1 410828.6 0.107 0.026 100.00 100.00
8GB simple numa-1-buffers colocated 437088.170225 438055.8 438324.7 438712.4 438931.0 439053.4 0.100 0.015 100.00 100.00
8GB simple numa-2-localalloc none 417427.169344 418270.3 418838.8 419372.3 420079.2 421378.3 0.105 0.012 100.00 100.00
8GB simple numa-2-localalloc random 407733.087076 409819.9 409989.2 410141.1 410349.4 410443.4 0.108 0.027 100.00 100.00
8GB simple numa-2-localalloc colocated 427076.145833 428141.6 428248.9 428421.9 428757.4 428942.7 0.103 0.016 100.00 100.00
8GB simple numa-3-no-tail none 416423.171947 417375.8 417944.6 418458.9 419220.5 419577.5 0.105 0.011 100.00 100.00
8GB simple numa-3-no-tail random 405829.718937 407883.1 408020.0 408131.0 408272.2 408496.8 0.108 0.026 100.00 100.00
8GB simple numa-3-no-tail colocated 421424.397567 422528.8 422703.8 422879.9 423100.0 423270.6 0.104 0.016 100.00 100.00
8GB simple numa-4-freelist none 419076.519833 420039.6 420611.5 421108.1 421851.3 422333.8 0.105 0.013 100.00 100.00
8GB simple numa-4-freelist random 399923.942600 402011.0 402182.1 402292.0 402462.0 402593.4 0.110 0.039 100.00 100.00
8GB simple numa-4-freelist colocated 408285.506814 409311.0 409527.0 409703.4 409848.1 409933.5 0.107 0.020 100.00 100.00
8GB simple numa-5-clocksweep none 418271.982316 419265.3 419777.1 420283.6 421177.6 421672.7 0.105 0.013 50.01 100.00
8GB simple numa-5-clocksweep random 412614.475308 414931.1 415027.7 415133.5 415266.1 415341.0 0.106 0.028 25.03 100.00
8GB simple numa-5-clocksweep colocated 413530.769663 414614.5 414791.1 414952.4 415109.0 415166.7 0.106 0.016 50.01 100.00
8GB simple numa-6-pgproc none 419625.446067 420662.8 421143.3 421698.9 422431.1 423015.8 0.105 0.013 25.02 100.00
8GB simple numa-6-pgproc random 413488.088807 415658.5 415842.1 415954.9 416118.8 416210.9 0.106 0.027 25.02 100.00
8GB simple numa-6-pgproc colocated 432241.958142 433413.7 433575.1 433732.2 434047.5 434135.9 0.101 0.015 50.01 100.00
8GB simple numa-7-pinning none 418205.981542 419096.8 419709.3 420246.4 421098.7 421900.0 0.105 0.013 25.02 100.00
8GB simple numa-7-pinning random 412783.906834 414853.9 415074.2 415432.1 415604.5 415712.6 0.106 0.027 50.01 100.00
8GB simple numa-7-pinning colocated 430539.555394 431708.8 431876.3 432020.8 432187.0 432274.7 0.102 0.017 25.02 100.00
8GB prepared numa-0-master none 703459.264267 704910.3 705975.2 707312.5 709332.2 710066.3 0.062 0.009 100.00 100.00
8GB prepared numa-0-master random 674244.112349 678564.5 678987.8 679454.4 680116.8 680501.2 0.065 0.025 100.00 100.00
8GB prepared numa-0-master colocated 714527.617635 716302.2 717010.7 717832.4 718839.7 719259.2 0.061 0.013 100.00 100.00
8GB prepared numa-1-buffers none 698173.827414 699533.4 700822.1 701923.8 703435.2 705671.6 0.063 0.009 100.00 100.00
8GB prepared numa-1-buffers random 673434.966316 677560.5 678049.7 678474.9 679062.0 679334.1 0.065 0.025 100.00 100.00
8GB prepared numa-1-buffers colocated 751566.117310 753320.4 754203.6 754831.3 755884.6 756464.8 0.058 0.012 100.00 100.00
8GB prepared numa-2-localalloc none 696928.357767 697891.6 699404.0 700916.8 702801.7 703614.1 0.063 0.009 100.00 100.00
8GB prepared numa-2-localalloc random 679125.597405 683401.6 683828.2 684343.2 684999.9 685223.4 0.064 0.024 100.00 100.00
8GB prepared numa-2-localalloc colocated 725088.057229 727100.1 727495.3 727966.1 728321.7 728666.5 0.060 0.011 100.00 100.00
8GB prepared numa-3-no-tail none 705127.816307 706515.8 707780.1 708851.5 710495.0 711812.6 0.062 0.009 100.00 100.00
8GB prepared numa-3-no-tail random 646426.356911 650349.4 650679.0 651084.4 651568.6 651890.3 0.068 0.025 100.00 100.00
8GB prepared numa-3-no-tail colocated 778274.192406 780341.2 780852.1 781509.3 782091.3 782312.0 0.056 0.012 100.00 100.00
8GB prepared numa-4-freelist none 688843.746004 690237.4 691476.0 692486.0 694485.6 695673.8 0.064 0.010 100.00 100.00
8GB prepared numa-4-freelist random 677409.346556 681680.9 682064.1 682463.7 683152.7 683382.7 0.065 0.029 100.00 100.00
8GB prepared numa-4-freelist colocated 777418.735444 779666.9 780072.2 780461.8 781045.9 781351.5 0.056 0.015 100.00 100.00
8GB prepared numa-5-clocksweep none 710909.011736 712640.0 713743.3 715092.0 716672.9 717912.9 0.062 0.010 50.01 100.00
8GB prepared numa-5-clocksweep random 693200.545485 697827.0 698333.0 698726.5 699376.2 699820.8 0.063 0.025 50.01 100.00
8GB prepared numa-5-clocksweep colocated 806920.631855 808679.8 809650.2 810959.6 811889.6 812648.1 0.054 0.013 25.02 100.00
8GB prepared numa-6-pgproc none 700719.229454 701968.1 703513.1 704830.2 706751.1 708149.9 0.063 0.010 25.02 100.00
8GB prepared numa-6-pgproc random 678244.443613 682195.4 682688.0 683070.6 683702.7 683965.3 0.065 0.025 25.02 100.00
8GB prepared numa-6-pgproc colocated 776707.680192 778630.0 779342.8 780372.5 781325.5 781939.7 0.056 0.013 25.02 100.00
8GB prepared numa-7-pinning none 707324.231529 708769.6 709673.6 710912.0 712442.2 713431.9 0.062 0.011 25.02 100.00
8GB prepared numa-7-pinning random 680872.484194 684928.7 685481.9 686003.1 686701.9 686972.7 0.064 0.025 25.02 100.00
8GB prepared numa-7-pinning colocated 784736.528912 786979.4 787586.1 788031.7 788616.5 788929.4 0.056 0.012 25.02 100.00
32GB simple numa-0-master none 481181.594955 485356.2 486575.9 487827.3 489003.9 489647.1 0.091 0.012 61.83 99.65
32GB simple numa-0-master random 466792.988100 472495.3 472962.0 473188.3 473459.6 473618.0 0.094 0.021 64.78 99.65
32GB simple numa-0-master colocated 482942.015715 486958.0 487531.7 488433.8 488592.6 488723.1 0.091 0.016 65.09 99.65
32GB simple numa-1-buffers none 480876.817874 485096.0 486160.2 486995.0 488304.7 489315.1 0.091 0.011 64.91 99.65
32GB simple numa-1-buffers random 460639.013021 466104.2 466494.0 466585.7 466694.7 466763.1 0.095 0.022 65.08 99.65
32GB simple numa-1-buffers colocated 488900.936303 493363.3 494035.3 494210.4 494426.5 494581.4 0.090 0.015 64.56 99.65
32GB simple numa-2-localalloc none 480484.228215 483794.7 485110.6 486269.1 487618.9 488064.9 0.091 0.011 64.68 99.65
32GB simple numa-2-localalloc random 459902.532232 464932.4 465444.2 465666.4 465843.8 465979.5 0.095 0.021 63.98 99.65
32GB simple numa-2-localalloc colocated 480102.371099 484327.4 484649.3 484841.3 485125.5 485420.4 0.091 0.014 63.97 99.65
32GB simple numa-3-no-tail none 471143.298591 474607.7 475852.4 476722.5 477940.3 478910.2 0.093 0.025 64.27 99.65
32GB simple numa-3-no-tail random 455567.453537 461096.7 461524.9 461848.0 462028.8 462150.5 0.096 0.022 64.21 99.65
32GB simple numa-3-no-tail colocated 491308.961289 495301.4 495628.7 495804.8 495992.0 496112.7 0.089 0.014 65.19 99.65
32GB simple numa-4-freelist none 480167.819014 483653.4 484628.6 485671.5 486689.1 487170.1 0.091 0.059 64.82 99.65
32GB simple numa-4-freelist random 472777.817626 478481.4 478987.1 479268.7 479542.4 479691.1 0.093 0.073 63.91 99.65
32GB simple numa-4-freelist colocated 492657.381699 496841.9 497470.7 497645.4 497820.0 497909.6 0.089 0.015 64.26 99.65
32GB simple numa-5-clocksweep none 483805.882520 489653.3 491572.4 492783.5 494432.2 495466.7 0.091 0.018 50.00 99.64
32GB simple numa-5-clocksweep random 468040.122668 474260.0 478357.7 478606.9 478963.5 479277.5 0.094 0.028 50.00 99.64
32GB simple numa-5-clocksweep colocated 482304.331109 486744.5 492023.4 492340.4 492606.3 492837.7 0.091 0.022 50.00 99.63
32GB simple numa-6-pgproc none 484479.447307 493097.9 494476.8 495464.5 496709.8 497211.6 0.090 0.027 25.00 99.64
32GB simple numa-6-pgproc random 457878.575808 464597.3 468862.4 469154.5 469344.3 469447.5 0.096 0.033 25.01 99.64
32GB simple numa-6-pgproc colocated 457805.057399 465111.0 467395.5 467654.0 467881.0 467959.6 0.096 0.028 25.01 99.64
32GB simple numa-7-pinning none 475346.501704 482907.0 485029.7 486132.5 487681.2 488656.5 0.092 0.024 25.00 99.64
32GB simple numa-7-pinning random 465628.120519 468806.0 476067.9 477194.0 477610.8 477786.0 0.094 0.029 50.00 99.64
32GB simple numa-7-pinning colocated 492061.233271 498352.0 501791.5 502016.0 502447.6 502844.3 0.089 0.023 50.00 99.64
32GB prepared numa-0-master none 883118.725149 884270.1 888725.8 891838.8 895229.1 897796.6 0.050 0.006 83.01 99.65
32GB prepared numa-0-master random 883127.213183 890679.6 891330.5 891699.5 892747.5 892965.9 0.050 0.014 83.94 99.65
32GB prepared numa-0-master colocated 1015416.796294 1019694.9 1020803.6 1023038.3 1023852.4 1024406.0 0.043 0.008 83.64 99.65
32GB prepared numa-1-buffers none 914220.121625 915460.5 918915.1 922621.8 926953.8 930621.5 0.048 0.005 83.77 99.65
32GB prepared numa-1-buffers random 871912.362629 879456.8 880058.4 881118.8 881811.5 882059.6 0.050 0.015 81.67 99.65
32GB prepared numa-1-buffers colocated 1049240.181634 1054167.4 1055629.3 1057257.7 1059329.6 1061070.9 0.042 0.008 82.07 99.65
32GB prepared numa-2-localalloc none 887616.565291 889474.7 892733.3 895617.5 900252.9 902337.3 0.049 0.005 82.84 99.65
32GB prepared numa-2-localalloc random 870508.549070 878103.2 878768.6 879038.5 879350.0 879493.6 0.050 0.015 82.16 99.65
32GB prepared numa-2-localalloc colocated 1020867.691158 1025611.1 1026682.9 1027615.3 1028681.4 1029457.2 0.043 0.008 82.23 99.65
32GB prepared numa-3-no-tail none 885775.797641 887798.4 891346.1 893870.0 897065.2 900503.7 0.049 0.005 83.08 99.65
32GB prepared numa-3-no-tail random 869874.940118 878303.4 879123.8 879556.0 880170.8 880301.7 0.050 0.015 81.38 99.65
32GB prepared numa-3-no-tail colocated 1023551.259812 1027846.4 1029726.4 1031542.5 1033057.1 1034226.2 0.043 0.008 82.18 99.65
32GB prepared numa-4-freelist none 888068.163937 890917.9 893654.9 896889.9 900299.0 903962.8 0.049 0.089 82.15 99.65
32GB prepared numa-4-freelist random 871501.689081 879536.5 880054.8 880596.9 881309.3 881502.9 0.050 0.054 81.62 99.65
32GB prepared numa-4-freelist colocated 1024148.485277 1030793.9 1031629.6 1032165.3 1032711.2 1033216.2 0.043 0.015 82.00 99.65
32GB prepared numa-5-clocksweep none 918069.797166 933014.9 936907.0 940382.8 944285.4 949159.3 0.048 0.018 25.00 99.64
32GB prepared numa-5-clocksweep random 885036.283920 894589.0 903520.2 905257.8 906320.4 906490.0 0.049 0.024 50.00 99.64
32GB prepared numa-5-clocksweep colocated 994162.961936 1011384.1 1018072.8 1020455.0 1021685.1 1022622.2 0.044 0.028 25.00 99.64
32GB prepared numa-6-pgproc none 907854.601956 919981.4 924082.5 928039.6 932036.9 934123.2 0.048 0.014 25.00 99.64
32GB prepared numa-6-pgproc random 879230.681109 894617.4 897742.0 898927.7 899484.6 899735.2 0.050 0.025 50.00 99.64
32GB prepared numa-6-pgproc colocated 1002350.405064 1016951.3 1022562.5 1023483.4 1025207.8 1025905.2 0.044 0.020 50.00 99.64
32GB prepared numa-7-pinning none 871253.230579 882872.7 885748.8 887650.3 890516.4 892967.4 0.050 0.014 50.00 99.64
32GB prepared numa-7-pinning random 846590.347022 854119.9 868664.9 874487.6 875205.1 875575.1 0.052 0.027 25.00 99.64
32GB prepared numa-7-pinning colocated 974530.487611 993520.8 995697.0 996657.9 998134.0 998447.0 0.045 0.022 26.91 99.64
[application/pdf] numa-xeon-e5-2699.pdf (53.6K, ../../[email protected]/12-numa-xeon-e5-2699.pdf)
download
[application/pdf] numa-hb176.pdf (54.2K, ../../[email protected]/13-numa-hb176.pdf)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-30 08:29 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-07-30 08:29 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Mon, Jul 28, 2025 at 4:22 PM Tomas Vondra <[email protected]> wrote:
Hi Tomas,
just a quick look here:
> 2) The PGPROC part introduces a similar registry, [..]
>
> There's also a view pg_buffercache_pgproc. The pg_buffercache location
> is a bit bogus - it has nothing to do with buffers, but it was good
> enough for now.
If you are looking for better names: pg_shmem_pgproc_numa would sound
like a more natural name.
> 3) The PGPROC partitioning is reworked and should fix the crash with the
> GUC set to "off".
Thanks!
> simple benchmark
> ----------------
[..]
> There's results for the three "pgbench pinning" strategies, and that can
> have pretty significant impact (colocated generally performs much better
> than either "none" or "random").
Hint: real world is that network cards are usually located on some PCI
slot that is assigned to certain node (so traffic is flowing from/to
there), so probably it would make some sense to put pgbench outside
this machine and remove this as "variable" anyway and remove the need
for that pgbench --pin-cpus in script. In optimal conditions: most
optimized layout would be probably to have 2 cards on separate PCI
slots, each for different node and some LACP between those, with
xmit_hash_policy allowing traffic distribution on both of those cards
-- usually there's not just single IP/MAC out there talking to/from
such server, so that would be real-world (or lack of) affinity.
Also classic pgbench workload, seems to be poor fit for testing it out
(at least v3-0001 buffers), there I would propose sticking to just
lots of big (~s_b size) full table seq scans to put stress on shared
memory. Classic pgbench is usually not there enough to put serious
bandwidth on the interconnect by my measurements.
> For the "bigger" machine (wiuth 176 cores) the incremental results look
> like this (for pinning=none, i.e. regular pgbench):
>
>
> mode s_b buffers localal no-tail freelist sweep pgproc pinning
> ====================================================================
> prepared 16GB 99% 101% 100% 103% 111% 99% 102%
> 32GB 98% 102% 99% 103% 107% 101% 112%
> 8GB 97% 102% 100% 102% 101% 101% 106%
> --------------------------------------------------------------------
> simple 16GB 100% 100% 99% 105% 108% 99% 108%
> 32GB 98% 101% 100% 103% 100% 101% 97%
> 8GB 100% 100% 101% 99% 100% 104% 104%
>
> The way I read this is that the first three patches have about no impact
> on throughput. Then freelist partitioning and (especially) clocksweep
> partitioning can help quite a bit. pgproc is again close to ~0%, and
> PGPROC pinning can help again (but this part is merely experimental).
Isn't the "pinning" column representing just numa_procs_pin=on ?
(shouldn't it be tested with numa_procs_interleave = on?)
[..]
> To quantify this kind of improvement, I think we'll need tests that
> intentionally cause (or try to) imbalance. If you have ideas for such
> tests, let me know.
Some ideas:
1. concurrent seq scans hitting s_b-sized table
2. one single giant PX-enabled seq scan with $VCPU workers (stresses
the importance of interleaving dynamic shm for workers)
3. select txid_current() with -M prepared?
> reserving number of huge pages
> ------------------------------
[..]
> It took me ages to realize what's happening, but it's very simple. The
> nr_hugepages is a global limit, but it's also translated into limits for
> each NUMA node. So when you write 16828 to it, in a 4-node system each
> node gets 1/4 of that. See
>
> $ numastat -cm
>
> Then we do the mmap(), and everything looks great, because there really
> is enough huge pages and the system can allocate memory from any NUMA
> node it needs.
Yup, similiar story as with OOMs just for per-zone/node.
> And then we come around, and do the numa_tonode_memory(). And that's
> where the issues start, because AFAIK this does not check the per-node
> limit of huge pages in any way. It just appears to work. And then later,
> when we finally touch the buffer, it tries to actually allocate the
> memory on the node, and realizes there's not enough huge pages. And
> triggers the SIGBUS.
I think that's why options for strict policy numa allocation exist and
I had the option to use it in my patches (anyway with one big call to
numa_interleave_memory() for everything it was much more simpler and
just not micromanaging things). Good reads are numa(3) but e.g.
mbind(2) underneath will tell you that e.g. `Before Linux 5.7.
MPOL_MF_STRICT was ignored on huge page mappings.` (I was on 6.14.x,
but it could be happening for you too if you start using it). Anyway,
numa_set_strict() is just wrapper around setting this exact flag
Anyway remember that volatile pg_numa_touch_mem_if_required()? - maybe
that should be always called in your patch series to pre-populate
everything during startup, so that others testing will get proper
guaranteed layout, even without issuing any pg_buffercache calls.
> The only way around this I found is by inflating the number of huge
> pages, significantly above the shared_memory_size_in_huge_pages value.
> Just to make sure the nodes get enough huge pages.
>
> I don't know what to do about this. It's quite annoying. If we only used
> huge pages for the partitioned parts, this wouldn't be a problem.
Meh, sacrificing a couple of huge pages (worst-case 1GB ?) just to get
NUMA affinity, seems like a logical trade-off, doesn't it?
But postgres -C shared_memory_size_in_huge_pages still works OK to
establish the exact count for vm.nr_hugepages, right?
Regards,
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-07-30 11:10 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-07-30 11:10 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 7/30/25 10:29, Jakub Wartak wrote:
> On Mon, Jul 28, 2025 at 4:22 PM Tomas Vondra <[email protected]> wrote:
>
> Hi Tomas,
>
> just a quick look here:
>
>> 2) The PGPROC part introduces a similar registry, [..]
>>
>> There's also a view pg_buffercache_pgproc. The pg_buffercache location
>> is a bit bogus - it has nothing to do with buffers, but it was good
>> enough for now.
>
> If you are looking for better names: pg_shmem_pgproc_numa would sound
> like a more natural name.
>
>> 3) The PGPROC partitioning is reworked and should fix the crash with the
>> GUC set to "off".
>
> Thanks!
>
>> simple benchmark
>> ----------------
> [..]
>> There's results for the three "pgbench pinning" strategies, and that can
>> have pretty significant impact (colocated generally performs much better
>> than either "none" or "random").
>
> Hint: real world is that network cards are usually located on some PCI
> slot that is assigned to certain node (so traffic is flowing from/to
> there), so probably it would make some sense to put pgbench outside
> this machine and remove this as "variable" anyway and remove the need
> for that pgbench --pin-cpus in script. In optimal conditions: most
> optimized layout would be probably to have 2 cards on separate PCI
> slots, each for different node and some LACP between those, with
> xmit_hash_policy allowing traffic distribution on both of those cards
> -- usually there's not just single IP/MAC out there talking to/from
> such server, so that would be real-world (or lack of) affinity.
>
The pgbench pinning certainly reduces some of the noise / overhead you
get when using multiple machines. I use it to "isolate" patches, and
make the effects more visible.
> Also classic pgbench workload, seems to be poor fit for testing it out
> (at least v3-0001 buffers), there I would propose sticking to just
> lots of big (~s_b size) full table seq scans to put stress on shared
> memory. Classic pgbench is usually not there enough to put serious
> bandwidth on the interconnect by my measurements.
>
Yes, that's possible. The simple pgbench workload is a bit of a "worst
case" for the NUMA patches, in that it's can benefit less from the
improvements, and it's also fairly sensitive to regressions.
I plan to do more tests with other types of workloads, like the one
doing a lot of large sequential scans, etc.
>> For the "bigger" machine (wiuth 176 cores) the incremental results look
>> like this (for pinning=none, i.e. regular pgbench):
>>
>>
>> mode s_b buffers localal no-tail freelist sweep pgproc pinning
>> ====================================================================
>> prepared 16GB 99% 101% 100% 103% 111% 99% 102%
>> 32GB 98% 102% 99% 103% 107% 101% 112%
>> 8GB 97% 102% 100% 102% 101% 101% 106%
>> --------------------------------------------------------------------
>> simple 16GB 100% 100% 99% 105% 108% 99% 108%
>> 32GB 98% 101% 100% 103% 100% 101% 97%
>> 8GB 100% 100% 101% 99% 100% 104% 104%
>>
>> The way I read this is that the first three patches have about no impact
>> on throughput. Then freelist partitioning and (especially) clocksweep
>> partitioning can help quite a bit. pgproc is again close to ~0%, and
>> PGPROC pinning can help again (but this part is merely experimental).
>
> Isn't the "pinning" column representing just numa_procs_pin=on ?
> (shouldn't it be tested with numa_procs_interleave = on?)
>
Maybe I don't understand the question, but the last column (pinning)
compares two builds.
1) Build with all the patches up to "pgproc interleaving" (and all of
the GUCs set to "on").
2) Build with all the patches from (1), and "pinning" too (again, all
GUCs set to "on).
Or do I misunderstand the question?
> [..]
>> To quantify this kind of improvement, I think we'll need tests that
>> intentionally cause (or try to) imbalance. If you have ideas for such
>> tests, let me know.
>
> Some ideas:
> 1. concurrent seq scans hitting s_b-sized table
> 2. one single giant PX-enabled seq scan with $VCPU workers (stresses
> the importance of interleaving dynamic shm for workers)
> 3. select txid_current() with -M prepared?
>
Thanks. I think we'll try something like (1), but it'll need to be a bit
more elaborate, because scans on tables larger than 1/4 shared buffers
use a small circular buffer.
>> reserving number of huge pages
>> ------------------------------
> [..]
>> It took me ages to realize what's happening, but it's very simple. The
>> nr_hugepages is a global limit, but it's also translated into limits for
>> each NUMA node. So when you write 16828 to it, in a 4-node system each
>> node gets 1/4 of that. See
>>
>> $ numastat -cm
>>
>> Then we do the mmap(), and everything looks great, because there really
>> is enough huge pages and the system can allocate memory from any NUMA
>> node it needs.
>
> Yup, similiar story as with OOMs just for per-zone/node.
>
>> And then we come around, and do the numa_tonode_memory(). And that's
>> where the issues start, because AFAIK this does not check the per-node
>> limit of huge pages in any way. It just appears to work. And then later,
>> when we finally touch the buffer, it tries to actually allocate the
>> memory on the node, and realizes there's not enough huge pages. And
>> triggers the SIGBUS.
>
> I think that's why options for strict policy numa allocation exist and
> I had the option to use it in my patches (anyway with one big call to
> numa_interleave_memory() for everything it was much more simpler and
> just not micromanaging things). Good reads are numa(3) but e.g.
> mbind(2) underneath will tell you that e.g. `Before Linux 5.7.
> MPOL_MF_STRICT was ignored on huge page mappings.` (I was on 6.14.x,
> but it could be happening for you too if you start using it). Anyway,
> numa_set_strict() is just wrapper around setting this exact flag
>
> Anyway remember that volatile pg_numa_touch_mem_if_required()? - maybe
> that should be always called in your patch series to pre-populate
> everything during startup, so that others testing will get proper
> guaranteed layout, even without issuing any pg_buffercache calls.
>
I think I tried using numa_set_strict, but it didn't change the behavior
(i.e. the numa_tonode_memory didn't error out).
>> The only way around this I found is by inflating the number of huge
>> pages, significantly above the shared_memory_size_in_huge_pages value.
>> Just to make sure the nodes get enough huge pages.
>>
>> I don't know what to do about this. It's quite annoying. If we only used
>> huge pages for the partitioned parts, this wouldn't be a problem.
>
> Meh, sacrificing a couple of huge pages (worst-case 1GB ?) just to get
> NUMA affinity, seems like a logical trade-off, doesn't it?
> But postgres -C shared_memory_size_in_huge_pages still works OK to
> establish the exact count for vm.nr_hugepages, right?
>
Well, yes and no. It tells you the exact number of huge pages, but it
does not tell you how much you need to inflate it to account for the
non-shared buffer part that may get allocated on a random node.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-04 14:24 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-08-04 14:24 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
Here's an updated version of the patch series. The main improvement is
the new 0006 patch, adding "adaptive balancing" of allocations. I'll
also share some results from a workload doing a lot of allocations.
adaptive balancing of allocations
---------------------------------
Imagine each backend only allocates buffers from the partition on the
same NUMA node. E.g. you have 4 NUMA nodes (i.e. 4 partitions), and a
backend only allocates buffers from "home" partition (on the same NUMA
node). This is what the earlier patch versions did, and with many
backends that's mostly fine (assuming the backends get spread over all
the NUMA nodes).
But if there's only few backends doing the allocations, this can result
in very inefficient use of shared buffers - a single backend would be
limited to 25% of buffers, even if the rest is unused.
There needs to be some say to "redirect" excess allocations to other
partitions, so that the partitions are utilized about the same. This is
what the 0006 patch aims to do (I kept is separate, but it should
probably get merged into the "clocksweep partitioning" in the end).
The balancing is fairly simple:
(1) It tracks the number of allocations "requested" from each partition.
(2) In regular intervals (by bgwriter) calculate the "fair share" per
partition, and determine what fraction of "requests" to handle from the
partition itself, and how many to redirect to other partitions.
(3) Calculate coefficients to drive this for each partition.
I emphasize (1) talks about "requests", not the actual allocations. Some
of the requests could have been redirected to different partitions, and
be counted as allocations there. We want to balance allocations, but it
relies on the requests.
To give you a simple example - imagine there are 2 partitions with this
number of allocation requests:
P1: 900,000 requests
P2: 100,000 requests
This means the "fair share" is 500,000 allocations, so P1 needs to
redirect some requests to P2. And we end with these weights:
P1: [ 55, 45]
P2: [ 0, 100]
Assuming the workload does not shift in some dramatic way, this should
result in both partitions handling ~500k allocations.
It's not hard to extend this algorithm to more partitions. For more
details see StrategySyncBalance(), which recalculates this.
There are a couple open questions, like:
* The algorithm combines the old/new weights by averaging, to add a bit
of hysteresis. Right now it's a simple average with 0.5 weight, to
dampen sudden changes. I think it works fine (in the long run), but I'm
open to suggestions how to do this better.
* There's probably additional things we should consider when deciding
where to redirect the allocations. For example, we may have multiple
partitions per NUMA node, in which case it's better to redirect to that
node as many allocations as possible. The current patch ignores this.
* The partitions may have slightly different sizes, but the balancing
ignores that for now. This is not very difficult to address.
clocksweep benchmark
--------------------
I ran a simple benchmark focused on allocation-heavy workloads, namely
large concurrent sequential scans. The attached scripts generate a
number of 1GB tables, and then run concurrent sequential scans with
shared buffers set to 60%, 75%, 90% and 110% of the total dataset size.
I did this for master, and with the NUMA patches applied (and the GUCs
set to 'on'). I also increased tried with the of partitions increased to
16 (so a NUMA node got multiple partitions).
There are results from three machines
1) ryzen - small non-NUMA system, mostly to see if there's regressions
2) xeon - older 2-node NUMA system
3) hb176 - big EPYC system with 176 cores / 4 NUMA nodes
The script records detailed TPS stats (e.g. percentiles), I'm attaching
CSV files with complete results, and some PDFs with charts summarizing
that (I'll get to that in a minute).
For the EPYC, the average tps for the three builds looks like this:
clients | master numa numa-16 | numa numa-16
----------|--------------------------------|---------------------
8 | 20 27 26 | 133% 129%
16 | 23 39 45 | 170% 193%
24 | 23 48 58 | 211% 252%
32 | 21 57 68 | 268% 321%
40 | 21 56 76 | 265% 363%
48 | 22 59 82 | 270% 375%
56 | 22 66 88 | 296% 397%
64 | 23 62 93 | 277% 411%
72 | 24 68 95 | 277% 389%
80 | 24 72 95 | 295% 391%
88 | 25 71 98 | 283% 392%
96 | 26 74 97 | 282% 369%
104 | 26 74 97 | 282% 367%
112 | 27 77 95 | 287% 355%
120 | 28 77 92 | 279% 335%
128 | 27 75 89 | 277% 328%
That's not bad - the clocksweep partitioning increases the throughput
2-3x. Having 16 partitions (instead of 4) helps yet a bit more, to 3-4x.
This is for shared buffers set to 60% of the dataset, which depends on
the number of clients / tables. With 64 clients/tables, there's 64GB of
data, and shared buffers are set to ~39GB.
The results for 75% and 90% follow the same pattern. For 110% there's
much less impact - there are no allocations, so this has to be thanks to
the other NUMA patches.
The charts in the attached PDFs add a bit more detail, with various
percentiles (of per-second throughput). The bands are roughly quartiles:
5-25%, 25-50%, 50-75%, 75-95%. The thick middle line is the median.
There's only charts for 60%, 90% and 110% shared buffers, for fit it on
a single page. There 75% is not very different.
For ryzen there's little difference. Not surprising, it's not a NUMA
system. So this is positive result, as there's no regression.
For xeon the patches help a little bit. Again, not surprising. It's a
fairly old system (~2016), and the differences between NUMA nodes are
not that significant.
For epyc (hb176), the differences are pretty massive.
regards
--
Tomas Vondra
Attachments:
[application/pdf] numa-benchmark-ryzen.pdf (162.5K, ../../[email protected]/2-numa-benchmark-ryzen.pdf)
download
[application/pdf] numa-benchmark-epyc.pdf (167.9K, ../../[email protected]/3-numa-benchmark-epyc.pdf)
download
[application/pdf] numa-benchmark-xeon.pdf (154.6K, ../../[email protected]/4-numa-benchmark-xeon.pdf)
download
[application/gzip] xeon.csv.gz (12.3K, ../../[email protected]/5-xeon.csv.gz)
download
[application/gzip] ryzen.csv.gz (12.0K, ../../[email protected]/6-ryzen.csv.gz)
download
[application/gzip] hb176.csv.gz (3.8K, ../../[email protected]/7-hb176.csv.gz)
download
[application/x-shellscript] run.sh (1.9K, ../../[email protected]/8-run.sh)
download
[application/x-shellscript] generate.sh (1.1K, ../../[email protected]/9-generate.sh)
download
[application/gzip] v20250804-0001-NUMA-interleaving-buffers.patch.gz (12.0K, ../../[email protected]/10-v20250804-0001-NUMA-interleaving-buffers.patch.gz)
download
[application/gzip] v20250804-0008-NUMA-pin-backends-to-NUMA-nodes.patch.gz (1.6K, ../../[email protected]/11-v20250804-0008-NUMA-pin-backends-to-NUMA-nodes.patch.gz)
download
[application/gzip] v20250804-0007-NUMA-interleave-PGPROC-entries.patch.gz (13.0K, ../../[email protected]/12-v20250804-0007-NUMA-interleave-PGPROC-entries.patch.gz)
download
[application/gzip] v20250804-0006-NUMA-clocksweep-allocation-balancing.patch.gz (7.6K, ../../[email protected]/13-v20250804-0006-NUMA-clocksweep-allocation-balancing.patch.gz)
download
[application/gzip] v20250804-0005-NUMA-clockweep-partitioning.patch.gz (10.4K, ../../[email protected]/14-v20250804-0005-NUMA-clockweep-partitioning.patch.gz)
download
[application/gzip] v20250804-0004-NUMA-partition-buffer-freelist.patch.gz (6.8K, ../../[email protected]/15-v20250804-0004-NUMA-partition-buffer-freelist.patch.gz)
download
[application/gzip] v20250804-0003-freelist-Don-t-track-tail-of-a-freelist.patch.gz (896B, ../../[email protected]/16-v20250804-0003-freelist-Don-t-track-tail-of-a-freelist.patch.gz)
download
[application/gzip] v20250804-0002-NUMA-localalloc.patch.gz (1.7K, ../../[email protected]/17-v20250804-0002-NUMA-localalloc.patch.gz)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-07 09:24 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 3 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-08-07 09:24 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi!
Here's a slightly improved version of the patch series.
The main improvement is related to rebalancing partitions of different
sizes (which can happen because the sizes have to be a multiple of some
minimal "chunk" determined by memory page size etc.). Part 0009 deals
with that by adjusting the allocations by partition size. It works OK,
but it's also true it matters less as the shared_buffers size increases
(as the relative difference between large/small partition gets smaller).
The other improvements are related to the pg_buffercache_partitions
view, showing the weights and (running) totals of allocations.
I plan to take a break from this patch series for a while, so this would
be a good time to take a look, do a review, run some tests etc. ;-)
One detail about the balancing I forgot to mention in my last message is
how the patch "distributes" allocations to match the balancing weights.
Consider for example the example weights from that message:
P1: [ 55, 45]
P2: [ 0, 100]
Imagine a backend located on P1 requests allocation of a buffer. The
weights say 55% buffers should be allocated from P1, and 45% should be
redirected to P2. One way to achieve that would be generating a random
number in [1, 100], and if it's [1,55] then P1, otherwise P2.
The patch does a much simpler thing - treat the weight as a "budget",
i.e. number of buffers to allocate before proceeding to the "next"
partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
and then goes back to P1 in a round-robin way. The advantage is it can
do away without a PRNG.
There's two things I'm not entirely sure about:
1) memory model - I'm not quite sure the current code ensures updates to
weights are properly "communicated" to the other processes. That is, if
the bgwriter recalculates the weights, will the other backends see the
new weights right away? Using a stale weights won't cause "failures",
the consequence is just a bit of imbalance. But it shouldn't stay like
that for too long, so maybe it'd be good to add some memory barriers or
something like that.
2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
assumes each core / piece of RAM is assigned to a particular NUMA node.
For the buffer partitioning the patch mostly cares about memory, as it
"locates" the buffers on different NUMA nodes. Which works mostly OK
(ignoring the issues with huge pages described in previous message).
But it also cares about the cores (and the node for each core), because
it uses that to pick the right partition for a backend. And here the
situation is less clear, because the CPUs don't need to be assigned to a
particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
$ numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3
node 0 size: 992 MB
node 0 free: 274 MB
node 1 cpus: 0 1 2 3
node 1 size: 1019 MB
node 1 free: 327 MB
node 2 cpus: 0 1 2 3
node 2 size: 1019 MB
node 2 free: 321 MB
node 3 cpus: 0 1 2 3
node 3 size: 955 MB
node 3 free: 251 MB
node 4 cpus: 0 1 2 3
node 4 size: 1019 MB
node 4 free: 332 MB
node 5 cpus: 0 1 2 3
node 5 size: 1019 MB
node 5 free: 342 MB
node 6 cpus: 0 1 2 3
node 6 size: 1019 MB
node 6 free: 352 MB
node 7 cpus: 0 1 2 3
node 7 size: 1014 MB
node 7 free: 339 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 10 10 10 10 10 10 10
1: 10 10 10 10 10 10 10 10
2: 10 10 10 10 10 10 10 10
3: 10 10 10 10 10 10 10 10
4: 10 10 10 10 10 10 10 10
5: 10 10 10 10 10 10 10 10
6: 10 10 10 10 10 10 10 10
7: 10 10 10 10 10 10 10 10
This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
are not assigned to particular nodes - each core is mapped to all 8 NUMA
nodes. I'm not sure what to do about this (or how getcpu() or libnuma
handle this). And can the situation be even more complicated?
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20250807-0011-NUMA-pin-backends-to-NUMA-nodes.patch (3.5K, ../../[email protected]/2-v20250807-0011-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From 630e4f0c9ce995ff5a9f1aaf68e8c64836cadffb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v20250807 11/11] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 11259151a7d..dbb4cbb1bfa 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -766,6 +766,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ee4684d1b8..3f88659b49f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7b718760248..862341e137e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2156,6 +2156,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index cdeee8dccba..a97741c6707 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v20250807-0010-NUMA-interleave-PGPROC-entries.patch (46.9K, ../../[email protected]/3-v20250807-0010-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From d2c7801011a394b5b9e00d60bf59e759b65e1bed Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v20250807 10/11] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 -
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 63 +-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 565 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 14 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 64 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 5acae31b836..ba54f69eeb4 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -38,3 +38,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 13014549d00..ee3aa8be2ce 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 15
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -946,3 +949,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..087279a6a8e 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 5b65a855b29..fb52039e1a6 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -500,8 +500,6 @@ buffer_partitions_prepare(void)
if (numa_nodes < 1)
numa_nodes = 1;
- elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
-
/*
* XXX A bit weird. Do we need to worry about postmaster? Could this even
* run outside postmaster? I don't think so.
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index bbe29bc9729..878d1e33f61 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -510,7 +510,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index bf987aed8d3..3e86e4ca2ae 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1622,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1739,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2223,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2297,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2498,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2724,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2755,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2857,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3019,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3060,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3188,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3231,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3300,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3402,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3467,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3522,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3568,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3597,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3628,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3669,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3732,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3798,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 62f3471448e..c84a2a5f1bc 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..11259151a7d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -90,6 +98,31 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static Size get_memory_page_size(void); /* XXX duplicate with bufi_init.c */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +133,63 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -129,6 +214,25 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -140,6 +244,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -191,7 +298,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -205,6 +312,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -241,19 +350,115 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partitoion
+ * per NUMA node.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += (numa_nodes * sizeof(PGProcPartition));
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /*
+ * Now initialize the PGPROC partition registry with a single
+ * partition for all the procs.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += sizeof(PGProcPartition);
+
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +491,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,15 +639,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +705,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2296,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2371,222 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(LOG, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(LOG, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (numa_procs_interleave && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a11bc71a386..6ee4684d1b8 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0552ed62cc7..7b718760248 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2146,6 +2146,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 66baf2bf33e..cdeee8dccba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..d2d269941fc 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -443,8 +448,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8540d537a3e..ded2db30422 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1876,6 +1876,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.50.1
[text/x-patch] v20250807-0009-NUMA-weighted-clocksweep-balancing.patch (5.1K, ../../[email protected]/4-v20250807-0009-NUMA-weighted-clocksweep-balancing.patch)
download | inline diff:
From 170673d4fe89bb9436c2af41216884fefa0a48ee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20250807 09/11] NUMA: weighted clocksweep balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index f2203cebcc8..bbe29bc9729 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -738,6 +738,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -764,16 +778,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -781,8 +806,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -845,6 +876,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -852,7 +887,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -866,22 +901,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.50.1
[text/x-patch] v20250807-0008-NUMA-clocksweep-allocation-balancing.patch (26.2K, ../../[email protected]/5-v20250807-0008-NUMA-clocksweep-allocation-balancing.patch)
download | inline diff:
From f832fedb847004990e7e806691ab4338752a5db5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 31 Jul 2025 19:50:05 +0200
Subject: [PATCH v20250807 08/11] NUMA: clocksweep allocation balancing
If backends only allocate buffers from the "local" partition, this could
cause significant misbalance - some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 41 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 396 ++++++++++++++++--
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 5 +-
6 files changed, 420 insertions(+), 31 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 999bb2128f0..5acae31b836 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -27,7 +27,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 7ca075e6164..13014549d00 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 15
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -789,6 +791,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -826,6 +830,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 12, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 13, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 14, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 15, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -851,11 +861,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -866,7 +882,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
&buffers_free,
&complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -904,6 +928,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[11] = Int64GetDatum(buffer_allocs);
nulls[11] = false;
+ values[12] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[12] = false;
+
+ values[13] = Int64GetDatum(buffer_req_allocs);
+ nulls[13] = false;
+
+ values[14] = PointerGetDatum(array);
+ nulls[14] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7a8c45ac59c..97b6f973c26 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3623,6 +3623,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 17988b4fd53..f2203cebcc8 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -51,6 +51,22 @@ typedef struct BufferStrategyFreelist
uint64 consumed;
} BufferStrategyFreelist;
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 16
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -83,9 +99,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -153,7 +188,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -165,7 +226,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -294,32 +355,72 @@ calculate_partition_index()
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
+
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
- return &StrategyControl->sweeps[index];
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
* ChooseFreeList
- * Pick the buffer freelist to use, depending on the CPU and NUMA node.
- *
- * Without partitioned freelists (numa_partition_freelist=false), there's only
- * a single freelist, so use that.
- *
- * With partitioned freelists, we have multiple ways how to pick the freelist
- * for the backend:
- *
- * - one freelist per CPU, use the freelist for CPU the task executes on
- *
- * - one freelist per NUMA node, use the freelist for node task executes on
+ * pick a clocksweep partition based on NUMA node and CPU
*
- * - use fixed number of freelists, map processes to lists based on PID
+ * The number of freelist partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
*
- * There may be some other strategies, not sure. The important thing is this
- * needs to be refrecled during initialization, i.e. we need to create the
- * right number of lists.
+ * XXX Maybe this should use the same balancing strategy as clocksweep?
*/
static BufferStrategyFreelist *
ChooseFreeList(void)
@@ -417,7 +518,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -580,6 +681,224 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- prepare for sync of all partitions
*
@@ -606,6 +925,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -852,7 +1172,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1242,7 +1576,9 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
BufferStrategyFreelist *freelist;
ClockSweep *sweep;
@@ -1288,11 +1624,21 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
/* get the clocksweep stats too */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 907b160b4f7..38bd5511048 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,6 +449,7 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index a6795c5fee9..e1729f0ee14 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -353,7 +353,10 @@ extern void FreelistPartitionGetInfo(int idx,
uint32 *complete_passes,
uint32 *next_victim_buffer,
uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.50.1
[text/x-patch] v20250807-0007-NUMA-clockweep-partitioning.patch (40.1K, ../../[email protected]/6-v20250807-0007-NUMA-clockweep-partitioning.patch)
download | inline diff:
From c2dbd991bf2720eb6d9295bffd61744f525a19c9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v20250807 07/11] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 476 ++++++++++--------
src/backend/storage/buffer/freelist.c | 238 +++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 6 +-
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 504 insertions(+), 261 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 95fd2d2a226..999bb2128f0 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -21,7 +21,13 @@ CREATE VIEW pg_buffercache_partitions AS
-- freelists
list_consumed bigint, -- buffers consumed from a freelist
list_remain bigint, -- buffers left in a freelist
- list_free bigint); -- number of free buffers
+ list_free bigint, -- number of free buffers
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 6d734464a22..7ca075e6164 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "list_free",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,6 +851,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -850,7 +864,9 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
- &buffers_free);
+ &buffers_free,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -876,6 +892,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffers_free);
nulls[7] = false;
+ values[8] = Int64GetDatum(complete_passes);
+ nulls[8] = false;
+
+ values[9] = Int32GetDatum(next_victim_buffer);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_total_allocs);
+ nulls[10] = false;
+
+ values[11] = Int64GetDatum(buffer_allocs);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index bd50535385f..7a8c45ac59c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3584,6 +3584,23 @@ BufferSync(int flags)
TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_scan);
}
+/*
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
+ *
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
+ *
+ * XXX might be better to have a per-partition struct with all the info
+ */
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
/*
* BgBufferSync -- Write out some dirty buffers in the pool.
*
@@ -3599,55 +3616,24 @@ bool
BgBufferSync(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
- int strategy_buf_id;
- uint32 strategy_passes;
uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
-
- /* Moving averages of allocation rate and clean-buffer density */
- static float smoothed_alloc = 0;
- static float smoothed_density = 10.0;
-
- /* Potentially these could be tunables, but for now, not */
- float smoothing_samples = 16;
- float scan_whole_pool_milliseconds = 120000.0;
-
- /* Used to compute how far we scan ahead */
- long strategy_delta;
- int bufs_to_lap;
- int bufs_ahead;
- float scans_per_alloc;
- int reusable_buffers_est;
- int upcoming_alloc_est;
- int min_scan_buffers;
-
- /* Variables for the scanning loop proper */
- int num_to_scan;
- int num_written;
- int reusable_buffers;
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
- /* Variables for final smoothed_density update */
- long new_strategy_delta;
- uint32 new_recent_alloc;
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
- /*
- * Find out where the freelist clock sweep currently is, and how many
- * buffer allocations have happened since our last call.
- */
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
/* Report buffer alloc counts to pgstat */
PendingBgWriterStats.buf_alloc += recent_alloc;
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -3660,223 +3646,285 @@ BgBufferSync(WritebackContext *wb_context)
}
/*
- * Compute strategy_delta = how many buffers have been scanned by the
- * clock sweep since last time. If first time through, assume none. Then
- * see if we are still ahead of the clock sweep, and if so, how many
- * buffers we could scan before we'd catch up with it and "lap" it. Note:
- * weird-looking coding of xxx_passes comparisons are to avoid bogus
- * behavior when the passes counts wrap around.
- */
- if (saved_info_valid)
- {
- int32 passes_delta = strategy_passes - prev_strategy_passes;
-
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* info obtained from freelist.c */
+ int strategy_buf_id;
+ uint32 strategy_passes;
+
+ /* Moving averages of allocation rate and clean-buffer density */
+ static float smoothed_alloc = 0;
+ static float smoothed_density = 10.0;
+
+ /* Potentially these could be tunables, but for now, not */
+ float smoothing_samples = 16;
+ float scan_whole_pool_milliseconds = 120000.0;
+
+ /* Used to compute how far we scan ahead */
+ long strategy_delta;
+ int bufs_to_lap;
+ int bufs_ahead;
+ float scans_per_alloc;
+ int reusable_buffers_est;
+ int upcoming_alloc_est;
+ int min_scan_buffers;
+
+ /* Variables for the scanning loop proper */
+ int num_to_scan;
+ int num_written;
+ int reusable_buffers;
+
+ /* Variables for final smoothed_density update */
+ long new_strategy_delta;
+ uint32 new_recent_alloc;
+
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
- Assert(strategy_delta >= 0);
+ /*
+ * Find out where the freelist clock sweep currently is, and how many
+ * buffer allocations have happened since our last call.
+ */
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
- if ((int32) (next_passes - strategy_passes) > 0)
+ /*
+ * Compute strategy_delta = how many buffers have been scanned by the
+ * clock sweep since last time. If first time through, assume none.
+ * Then see if we are still ahead of the clock sweep, and if so, how
+ * many buffers we could scan before we'd catch up with it and "lap"
+ * it. Note: weird-looking coding of xxx_passes comparisons are to
+ * avoid bogus behavior when the passes counts wrap around.
+ */
+ if (saved_info_valid)
{
- /* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
+
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
+
+ Assert(strategy_delta >= 0);
+
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
+ {
+ /* we're one pass ahead of the strategy point */
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
#endif
- }
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
- {
- /* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ }
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
+ {
+ /* on same pass, but ahead or at least not behind */
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
+#ifdef BGW_DEBUG
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
+#endif
+ }
+ else
+ {
+ /*
+ * We're behind, so skip forward to the strategy point and
+ * start cleaning from there.
+ */
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta);
#endif
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
+ }
}
else
{
/*
- * We're behind, so skip forward to the strategy point and start
- * cleaning from there.
+ * Initializing at startup or after LRU scanning had been off.
+ * Always start at the strategy point.
*/
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta);
+ elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
+ strategy_passes, strategy_buf_id);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ strategy_delta = 0;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
- }
- else
- {
- /*
- * Initializing at startup or after LRU scanning had been off. Always
- * start at the strategy point.
- */
-#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
- strategy_passes, strategy_buf_id);
-#endif
- strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
- }
- /* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ /* Update saved info for next time */
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* FIXME has to happen after all partitions */
+ /* saved_info_valid = true; */
- /*
- * Compute how many buffers had to be scanned for each new allocation, ie,
- * 1/density of reusable buffers, and track a moving average of that.
- *
- * If the strategy point didn't move, we don't update the density estimate
- */
- if (strategy_delta > 0 && recent_alloc > 0)
- {
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
- }
+ /*
+ * Compute how many buffers had to be scanned for each new allocation,
+ * ie, 1/density of reusable buffers, and track a moving average of
+ * that.
+ *
+ * If the strategy point didn't move, we don't update the density
+ * estimate
+ */
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
+ {
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
+ }
- /*
- * Estimate how many reusable buffers there are between the current
- * strategy point and where we've scanned ahead to, based on the smoothed
- * density estimate.
- */
- bufs_ahead = NBuffers - bufs_to_lap;
- reusable_buffers_est = (float) bufs_ahead / smoothed_density;
+ /*
+ * Estimate how many reusable buffers there are between the current
+ * strategy point and where we've scanned ahead to, based on the
+ * smoothed density estimate.
+ */
+ bufs_ahead = num_buffers - bufs_to_lap;
+ reusable_buffers_est = (float) bufs_ahead / smoothed_density;
- /*
- * Track a moving average of recent buffer allocations. Here, rather than
- * a true average we want a fast-attack, slow-decline behavior: we
- * immediately follow any increase.
- */
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
- else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
- smoothing_samples;
+ /*
+ * Track a moving average of recent buffer allocations. Here, rather
+ * than a true average we want a fast-attack, slow-decline behavior:
+ * we immediately follow any increase.
+ */
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
+ else
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
+ smoothing_samples;
- /* Scale the estimate by a GUC to allow more aggressive tuning. */
- upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
+ /* Scale the estimate by a GUC to allow more aggressive tuning. */
+ upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
- /*
- * If recent_alloc remains at zero for many cycles, smoothed_alloc will
- * eventually underflow to zero, and the underflows produce annoying
- * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
- * zero, there's no point in tracking smaller and smaller values of
- * smoothed_alloc, so just reset it to exactly zero to avoid this
- * syndrome. It will pop back up as soon as recent_alloc increases.
- */
- if (upcoming_alloc_est == 0)
- smoothed_alloc = 0;
+ /*
+ * If recent_alloc remains at zero for many cycles, smoothed_alloc
+ * will eventually underflow to zero, and the underflows produce
+ * annoying kernel warnings on some platforms. Once
+ * upcoming_alloc_est has gone to zero, there's no point in tracking
+ * smaller and smaller values of smoothed_alloc, so just reset it to
+ * exactly zero to avoid this syndrome. It will pop back up as soon
+ * as recent_alloc increases.
+ */
+ if (upcoming_alloc_est == 0)
+ smoothed_alloc = 0;
- /*
- * Even in cases where there's been little or no buffer allocation
- * activity, we want to make a small amount of progress through the buffer
- * cache so that as many reusable buffers as possible are clean after an
- * idle period.
- *
- * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
- * the BGW will be called during the scan_whole_pool time; slice the
- * buffer pool into that many sections.
- */
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ /*
+ * Even in cases where there's been little or no buffer allocation
+ * activity, we want to make a small amount of progress through the
+ * buffer cache so that as many reusable buffers as possible are clean
+ * after an idle period.
+ *
+ * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many
+ * times the BGW will be called during the scan_whole_pool time; slice
+ * the buffer pool into that many sections.
+ */
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
- if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
- {
+ if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
+ {
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
- upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
+ elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
+ upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
#endif
- upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
- }
-
- /*
- * Now write out dirty reusable buffers, working forward from the
- * next_to_clean point, until we have lapped the strategy scan, or cleaned
- * enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.
- */
+ upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
+ }
- num_to_scan = bufs_to_lap;
- num_written = 0;
- reusable_buffers = reusable_buffers_est;
+ /*
+ * Now write out dirty reusable buffers, working forward from the
+ * next_to_clean point, until we have lapped the strategy scan, or
+ * cleaned enough buffers to match our estimate of the next cycle's
+ * allocation requirements, or hit the bgwriter_lru_maxpages limit.
+ */
- /* Execute the LRU scan */
- while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
- {
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ num_to_scan = bufs_to_lap;
+ num_written = 0;
+ reusable_buffers = reusable_buffers_est;
- if (++next_to_clean >= NBuffers)
+ /* Execute the LRU scan */
+ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- next_to_clean = 0;
- next_passes++;
- }
- num_to_scan--;
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
+ wb_context);
- if (sync_state & BUF_WRITTEN)
- {
- reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- PendingBgWriterStats.maxwritten_clean++;
- break;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
+ }
+ num_to_scan--;
+
+ if (sync_state & BUF_WRITTEN)
+ {
+ reusable_buffers++;
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
+ {
+ PendingBgWriterStats.maxwritten_clean++;
+ break;
+ }
}
+ else if (sync_state & BUF_REUSABLE)
+ reusable_buffers++;
}
- else if (sync_state & BUF_REUSABLE)
- reusable_buffers++;
- }
- PendingBgWriterStats.buf_written_clean += num_written;
+ PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
- elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
- smoothed_density, reusable_buffers_est, upcoming_alloc_est,
- bufs_to_lap - num_to_scan,
- num_written,
- reusable_buffers - reusable_buffers_est);
+ elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
+ smoothed_density, reusable_buffers_est, upcoming_alloc_est,
+ bufs_to_lap - num_to_scan,
+ num_written,
+ reusable_buffers - reusable_buffers_est);
#endif
- /*
- * Consider the above scan as being like a new allocation scan.
- * Characterize its density and update the smoothed one based on it. This
- * effectively halves the moving average period in cases where both the
- * strategy and the background writer are doing some useful scanning,
- * which is helpful because a long memory isn't as desirable on the
- * density estimates.
- */
- new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
- {
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
+ /*
+ * Consider the above scan as being like a new allocation scan.
+ * Characterize its density and update the smoothed one based on it.
+ * This effectively halves the moving average period in cases where
+ * both the strategy and the background writer are doing some useful
+ * scanning, which is helpful because a long memory isn't as desirable
+ * on the density estimates.
+ */
+ new_strategy_delta = bufs_to_lap - num_to_scan;
+ new_recent_alloc = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ {
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
- scans_per_alloc, smoothed_density);
+ elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
+ new_recent_alloc, new_strategy_delta,
+ scans_per_alloc, smoothed_density);
#endif
+ }
+
+ /* hibernate if all partitions can hibernate */
+ hibernate &= (bufs_to_lap == 0 && recent_alloc_partition == 0);
}
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 5a63dad7f2c..17988b4fd53 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -52,17 +52,27 @@ typedef struct BufferStrategyFreelist
} BufferStrategyFreelist;
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* Clock sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -73,6 +83,19 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
@@ -88,6 +111,9 @@ typedef struct
int num_partitions;
int num_partitions_per_node;
+ /* clocksweep partitions */
+ ClockSweep *sweeps;
+
BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
@@ -127,6 +153,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -138,6 +165,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -145,14 +173,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -178,19 +206,23 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
}
static int
@@ -247,6 +279,28 @@ calculate_partition_index()
return index;
}
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
+}
+
/*
* ChooseFreeList
* Pick the buffer freelist to use, depending on the CPU and NUMA node.
@@ -363,7 +417,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -434,13 +488,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Nothing on the freelist, so run the "clock sweep" algorithm
*
- * XXX Should we also make this NUMA-aware, to only access buffers from
- * the same NUMA node? That'd probably mean we need to make the clock
- * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
- * subset of buffers. But that also means each process could "sweep" only
- * a fraction of buffers, even if the other buffers are better candidates
- * for eviction. Would that also mean we'd have multiple bgwriters, one
- * for each node, or would one bgwriter handle all of that?
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
*/
trycounter = NBuffers;
for (;;)
@@ -522,6 +580,46 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncStart -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -529,37 +627,44 @@ StrategyFreeBuffer(BufferDesc *buf)
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -647,6 +752,10 @@ StrategyShmemSize(void)
size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
num_partitions)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -665,6 +774,7 @@ StrategyInitialize(bool init)
int num_nodes;
int num_partitions;
int num_partitions_per_node;
+ char *ptr;
/* */
BufferPartitionParams(&num_partitions, &num_nodes);
@@ -692,7 +802,8 @@ StrategyInitialize(bool init)
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
- MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -707,12 +818,42 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* have to point the sweeps array to right after the freelists */
+ ptr = (char *) StrategyControl +
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions);
+ StrategyControl->sweeps = (ClockSweep *) ptr;
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -760,7 +901,6 @@ StrategyInitialize(bool init)
buf->freeNext = freelist->firstFreeBuffer;
freelist->firstFreeBuffer = i;
}
-
}
}
else
@@ -1100,9 +1240,12 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
}
void
-FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
{
BufferStrategyFreelist *freelist;
+ ClockSweep *sweep;
int cur;
/* stats */
@@ -1112,6 +1255,7 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
freelist = &StrategyControl->freelists[idx];
+ sweep = &StrategyControl->sweeps[idx];
/* stat */
SpinLockAcquire(&freelist->freelist_lock);
@@ -1141,4 +1285,14 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
*remain = cnt_remain;
*actually_free = cnt_free;
+
+ /* get the clocksweep stats too */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9dfbecb9fe4..907b160b4f7 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,7 +449,9 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index df127274190..a6795c5fee9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -349,7 +349,11 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void FreelistPartitionGetInfo(int idx,
uint64 *consumed, uint64 *remain,
- uint64 *actually_free);
+ uint64 *actually_free,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 03ca3b7c8bc..8540d537a3e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -427,6 +427,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.50.1
[text/x-patch] v20250807-0006-NUMA-partition-buffer-freelist.patch (20.5K, ../../[email protected]/7-v20250807-0006-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From a505c4e23d60d5aac911c24391444ba1b8320bf0 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v20250807 06/11] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
This uses the buffer partitioning scheme introduced by the earlier
patch, i.e. the partitions will "align" with NUMA nodes, etc.
It also extends the "pg_buffercache_partitions" view, to include
information about each freelist (number of consumedd buffers, ...).
When allocating a buffer, it's taken from the correct freelist (same
NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
.../pg_buffercache--1.6--1.7.sql | 7 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/freelist.c | 349 ++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 4 +-
7 files changed, 366 insertions(+), 30 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index fb9003c011e..95fd2d2a226 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -16,7 +16,12 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- freelists
+ list_consumed bigint, -- buffers consumed from a freelist
+ list_remain bigint, -- buffers left in a freelist
+ list_free bigint); -- number of free buffers
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8baa7c7b543..6d734464a22 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -812,6 +812,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "list_consumed",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "list_remain",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "list_free",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -833,12 +839,19 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffers_consumed,
+ buffers_remain,
+ buffers_free;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
+ &buffers_free);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -854,6 +867,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(buffers_consumed);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffers_remain);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffers_free);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..5a63dad7f2c 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,41 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
/*
* The shared freelist control information.
@@ -39,8 +66,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,8 +76,19 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -157,6 +193,88 @@ ClockSweepTick(void)
return victim;
}
+static int
+calculate_partition_index()
+{
+ int rc;
+ unsigned cpu;
+ unsigned node;
+ int index;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->freelists[index];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +286,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +314,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +381,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +431,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -356,7 +492,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ /*
+ * We don't want to call ChooseFreeList() again, because we might get a
+ * completely different freelist - either a different partition in the
+ * same group, or even a different group if the NUMA node changed. But we
+ * can calculate the proper freelist from the buffer id.
+ */
+ int index = (BufferGetNode(buf->buf_id) * StrategyControl->num_partitions_per_node)
+ + (buf->buf_id % StrategyControl->num_partitions_per_node);
+
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[index];
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +515,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +583,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int p = 0; p < StrategyControl->num_partitions; p++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[p];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "NUMA: freelist partition %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ p,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -445,12 +632,20 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+ int num_nodes;
+
+ BufferPartitionParams(&num_partitions, &num_nodes);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /* size of freelist partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ num_partitions)));
return size;
}
@@ -467,6 +662,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* */
+ BufferPartitionParams(&num_partitions, &num_nodes);
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -484,7 +691,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
&found);
if (!found)
@@ -494,13 +702,10 @@ StrategyInitialize(bool init)
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +716,52 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < num_partitions; nfreelist++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(nfreelist, &node,
+ &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * Walk through buffers for each partition, add them to the list.
+ * Walk from the end, because we're adding the buffers to the
+ * beginning.
+ */
+
+ for (int i = last_buffer; i >= first_buffer; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+
+ /* add to the freelist */
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
+
+ }
}
else
Assert(!init);
@@ -847,3 +1098,47 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+{
+ BufferStrategyFreelist *freelist;
+ int cur;
+
+ /* stats */
+ uint64 cnt_remain = 0;
+ uint64 cnt_free = 0;
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[idx];
+
+ /* stat */
+ SpinLockAcquire(&freelist->freelist_lock);
+
+ *consumed = freelist->consumed;
+
+ cur = freelist->firstFreeBuffer;
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ cnt_remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ cnt_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ SpinLockRelease(&freelist->freelist_lock);
+
+ *remain = cnt_remain;
+ *actually_free = cnt_free;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..a11bc71a386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+bool numa_partition_freelist = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a21f20800fb..0552ed62cc7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2136,6 +2136,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..66baf2bf33e 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index deaf4f19fa4..df127274190 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -347,7 +347,9 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
-
+extern void FreelistPartitionGetInfo(int idx,
+ uint64 *consumed, uint64 *remain,
+ uint64 *actually_free);
/* inline functions */
--
2.50.1
[text/x-patch] v20250807-0005-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../../[email protected]/8-v20250807-0005-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From e0fa771531b92dc096fd5a1580ded3224524fa65 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v20250807 05/11] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.50.1
[text/x-patch] v20250807-0004-NUMA-localalloc.patch (3.7K, ../../[email protected]/9-v20250807-0004-NUMA-localalloc.patch)
download | inline diff:
From 7fec3b33652e92903a20276536477df33be722c8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v20250807 04/11] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 17 +++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 29 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 65d8cbfaed5..d986a1d18cf 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,19 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9570087aa60..a21f20800fb 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v20250807-0003-NUMA-interleaving-buffers.patch (38.4K, ../../[email protected]/10-v20250807-0003-NUMA-interleaving-buffers.patch)
download | inline diff:
From bbe6427b4e0e871bcb7b2cc4ce11ad8aba62799c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 28 Jul 2025 14:01:37 +0200
Subject: [PATCH v20250807 03/11] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to 'off'), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by numa_buffers_interleave GUC (default: false)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
contrib/pg_buffercache/Makefile | 2 +-
.../pg_buffercache--1.6--1.7.sql | 26 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 626 +++++++++++++++++-
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 2 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 15 +
src/tools/pgindent/typedefs.list | 2 +
11 files changed, 775 insertions(+), 11 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..fb9003c011e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,26 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae0291e6e96..8baa7c7b543 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -771,3 +773,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..5b65a855b29 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +32,19 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsArray;
+
+static Size get_memory_page_size(void);
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
@@ -70,19 +91,89 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +203,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize the registry of buffer partitions, and also move the
+ * memory to different NUMA nodes (if enabled by GUC)
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +241,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +254,68 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -184,5 +330,467 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+
+#define MIN_BUFFER_PARTITIONS 4
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(LOG, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (numa_partitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ part->numa_node = n;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(LOG, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (!numa_buffers_interleave || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ *num_partitions = numa_partitions;
+ *num_nodes = numa_nodes;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9570087aa60 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 52a71b138f7..9dfbecb9fe4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -323,6 +323,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -491,4 +492,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..deaf4f19fa4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -143,6 +143,20 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+typedef struct BufferPartition
+{
+ int numa_node;
+ int num_buffers;
+ int first_buffer;
+ int last_buffer;
+} BufferPartition;
+
+typedef struct BufferPartitions
+{
+ int npartitions;
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* forward declared, to avoid having to expose buf_internals.h here */
struct WritebackContext;
@@ -319,6 +333,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e6f2e93b2d6..03ca3b7c8bc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.50.1
[text/x-patch] v20250807-0002-nbtree-Use-ReadRecentBuffer-in_bt_getroot.patch (7.3K, ../../[email protected]/11-v20250807-0002-nbtree-Use-ReadRecentBuffer-in_bt_getroot.patch)
download | inline diff:
From 730467cfcf1d1b7f0a22f61bd48a37371f80cce9 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 27 Jun 2023 14:07:34 -0700
Subject: [PATCH v20250807 02/11] nbtree: Use ReadRecentBuffer()
in_bt_getroot()
We tend to access the btree root page over and over, when descending the
index. It's quite expensive. Not really specific to NUMA, but it hurts
there much more.
Thomas Munro worked on this.
Discussion: https://www.postgresql.org/message-id/20230627020546.t6z4tntmj7wmjrfh%40awork3.anarazel.de
Discussion: https://www.postgresql.org/message-id/CA%2BhUKGJ8N_DRSB0YioinWjS2ycMpmOLy32mbBqVVztwBvXgyJA%40mail.gmail.com
---
src/backend/access/nbtree/nbtpage.c | 21 ++++++++++++++--
src/backend/access/nbtree/nbtsearch.c | 17 +++++++++----
src/backend/storage/buffer/bufmgr.c | 35 +++++----------------------
src/include/utils/backend_status.h | 4 +++
src/include/utils/rel.h | 4 +++
5 files changed, 45 insertions(+), 36 deletions(-)
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index c79dd38ee18..3cc26490f06 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -35,6 +35,7 @@
#include "storage/procarray.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/snapmgr.h"
static BTMetaPageData *_bt_getmeta(Relation rel, Buffer metabuf);
@@ -344,7 +345,7 @@ Buffer
_bt_getroot(Relation rel, Relation heaprel, int access)
{
Buffer metabuf;
- Buffer rootbuf;
+ Buffer rootbuf = InvalidBuffer;
Page rootpage;
BTPageOpaque rootopaque;
BlockNumber rootblkno;
@@ -373,7 +374,20 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
Assert(rootblkno != P_NONE);
rootlevel = metad->btm_fastlevel;
- rootbuf = _bt_getbuf(rel, rootblkno, BT_READ);
+
+ if (BufferIsValid(rel->rd_recent_root))
+ {
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, rootblkno,
+ rel->rd_recent_root))
+ {
+ rootbuf = rel->rd_recent_root;
+ _bt_lockbuf(rel, rootbuf, BT_READ);
+ _bt_checkpage(rel, rootbuf);
+ }
+ }
+
+ if (rootbuf == InvalidBuffer)
+ rootbuf = _bt_getbuf(rel, rootblkno, BT_READ);
rootpage = BufferGetPage(rootbuf);
rootopaque = BTPageGetOpaque(rootpage);
@@ -390,6 +404,7 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
P_RIGHTMOST(rootopaque))
{
/* OK, accept cached page as the root */
+ rel->rd_recent_root = rootbuf;
return rootbuf;
}
_bt_relbuf(rel, rootbuf);
@@ -555,6 +570,8 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
rootopaque->btpo_level, rootlevel);
}
+ rel->rd_recent_root = rootbuf;
+
/*
* By here, we have a pin and read lock on the root page, and no lock set
* on the metadata page. Return the root page's buffer.
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index d69798795b4..e1e73b4aaf3 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -168,11 +168,17 @@ _bt_search(Relation rel, Relation heaprel, BTScanInsert key, Buffer *bufP,
* stack entry for this page/level. If caller ends up splitting a
* page one level down, it usually ends up inserting a new pivot
* tuple/downlink immediately after the location recorded here.
+ *
+ * FIXME: Unfortunately this isn't a usable gating condition, as
+ * vacuum uses BT_READ and needs the stack.
*/
- new_stack = (BTStack) palloc(sizeof(BTStackData));
- new_stack->bts_blkno = BufferGetBlockNumber(*bufP);
- new_stack->bts_offset = offnum;
- new_stack->bts_parent = stack_in;
+ if (false && access == BT_WRITE)
+ {
+ new_stack = (BTStack) palloc(sizeof(BTStackData));
+ new_stack->bts_blkno = BufferGetBlockNumber(*bufP);
+ new_stack->bts_offset = offnum;
+ new_stack->bts_parent = stack_in;
+ }
/*
* Page level 1 is lowest non-leaf page level prior to leaves. So, if
@@ -186,7 +192,8 @@ _bt_search(Relation rel, Relation heaprel, BTScanInsert key, Buffer *bufP,
*bufP = _bt_relandgetbuf(rel, *bufP, child, page_access);
/* okay, all set to move down a level */
- stack_in = new_stack;
+ if (false && access == BT_WRITE)
+ stack_in = new_stack;
}
/*
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 67431208e7f..bd50535385f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -685,7 +685,6 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
BufferDesc *bufHdr;
BufferTag tag;
uint32 buf_state;
- bool have_private_ref;
Assert(BufferIsValid(recent_buffer));
@@ -713,38 +712,16 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
else
{
bufHdr = GetBufferDescriptor(recent_buffer - 1);
- have_private_ref = GetPrivateRefCount(recent_buffer) > 0;
- /*
- * Do we already have this buffer pinned with a private reference? If
- * so, it must be valid and it is safe to check the tag without
- * locking. If not, we have to lock the header first and then check.
- */
- if (have_private_ref)
- buf_state = pg_atomic_read_u32(&bufHdr->state);
- else
- buf_state = LockBufHdr(bufHdr);
-
- if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag))
+ if (!PinBuffer(bufHdr, NULL) ||
+ !BufferTagsEqual(&tag, &bufHdr->tag))
{
- /*
- * It's now safe to pin the buffer. We can't pin first and ask
- * questions later, because it might confuse code paths like
- * InvalidateBuffer() if we pinned a random non-matching buffer.
- */
- if (have_private_ref)
- PinBuffer(bufHdr, NULL); /* bump pin count */
- else
- PinBuffer_Locked(bufHdr); /* pin for first time */
-
- pgBufferUsage.shared_blks_hit++;
-
- return true;
+ UnpinBuffer(bufHdr);
+ return false;
}
- /* If we locked the header above, now unlock. */
- if (!have_private_ref)
- UnlockBufHdr(bufHdr, buf_state);
+ pgBufferUsage.shared_blks_hit++;
+ return true;
}
return false;
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 3016501ac05..443c492cf34 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -97,6 +97,10 @@ typedef struct PgBackendGSSStatus
*/
typedef struct PgBackendStatus
{
+#ifdef pg_attribute_aligned
+ pg_attribute_aligned(PG_CACHE_LINE_SIZE)
+#endif
+
/*
* To avoid locking overhead, we use the following protocol: a backend
* increments st_changecount before modifying its entry, and again after
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b552359915f..7ac2fb1512f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -25,6 +25,7 @@
#include "rewrite/prs2lock.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
+#include "storage/buf.h"
#include "storage/smgr.h"
#include "utils/relcache.h"
#include "utils/reltrigger.h"
@@ -159,6 +160,9 @@ typedef struct RelationData
/* data managed by RelationGetIndexAttrBitmap: */
bool rd_attrsvalid; /* are bitmaps of attrs valid? */
+
+ Buffer rd_recent_root;
+
Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */
Bitmapset *rd_pkattr; /* cols included in primary key */
Bitmapset *rd_idattr; /* included in replica identity index */
--
2.50.1
[text/x-patch] v20250807-0001-allow-pgbench-to-pin-backends-threads.patch (11.6K, ../../[email protected]/12-v20250807-0001-allow-pgbench-to-pin-backends-threads.patch)
download | inline diff:
From 9097f04069ab604146606a7f6eabe9a2ea78dca0 Mon Sep 17 00:00:00 2001
From: Ubuntu
<azureuser@hb176.osdv23jxzvxutpvofmy4uk5hcd.bx.internal.cloudapp.net>
Date: Sat, 31 May 2025 00:45:47 +0000
Subject: [PATCH v20250807 01/11] allow pgbench to pin backends/threads
Supported policies are "none" (default), "colocated" (both processes on
the same random core) and "random" (both processes on different cores).
---
src/bin/pgbench/pgbench.c | 348 ++++++++++++++++++++++++++++++++++++++
1 file changed, 348 insertions(+)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 125f3c7bbbe..95479861177 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -41,6 +41,7 @@
#include <time.h>
#include <sys/time.h>
#include <sys/resource.h> /* for getrlimit */
+#include <sys/sysinfo.h>
/* For testing, PGBENCH_USE_SELECT can be defined to force use of that code */
#if defined(HAVE_PPOLL) && !defined(PGBENCH_USE_SELECT)
@@ -300,6 +301,12 @@ static const char *dbName = NULL;
static char *logfile_prefix = NULL;
static const char *progname;
+#define CPU_PINNING_NONE 0
+#define CPU_PINNING_RANDOM 1
+#define CPU_PINNING_COLOCATED 2
+
+static int pinning_mode = CPU_PINNING_NONE;
+
#define WSEP '@' /* weight separator */
static volatile sig_atomic_t timer_exceeded = false; /* flag from signal
@@ -639,6 +646,8 @@ typedef struct
int64 cnt; /* client transaction count, for -t; skipped
* and failed transactions are also counted
* here */
+
+ int cpu; /* CPU to pin to, -1 = no pinning */
} CState;
/*
@@ -672,6 +681,8 @@ typedef struct
StatsData stats;
int64 latency_late; /* count executed but late transactions */
+
+ int cpu; /* CPU to pin to, -1 = no pinning */
} TState;
/*
@@ -841,6 +852,45 @@ static void add_socket_to_set(socket_set *sa, int fd, int idx);
static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
+/*
+ * random number generator for assigning CPUs to processes/threads
+ *
+ * This is not a simple generator, because we want to keep the outcome
+ * balanced - it's not just about the total number of threads/processes
+ * assigned to each core, because a thread is likely much lighter than a
+ * backend. So we want to distribute each of those uniformly.
+ *
+ * For example, assume we have 2 cores, and want to pin 2 threads and 2
+ * backends. We could pin 2 threads to core 0 and 2 backends to core 1.
+ * But that would not be balanced, because core 1 will have to execute
+ * much more work. Instead, we want to assign 1+1 to each core.
+ *
+ * Another restriction is that we don't want to assign both sides of
+ * a connection to the same core.
+ *
+ * All of that however applies to the 'random' mode. In the 'colocated'
+ * mode, we want to assign both sides of a connection to the same core.
+ * But we still pick the core randomly.
+ */
+typedef struct cpu_generator_state
+{
+ int ncpus; /* number of CPUs available */
+ int nitems; /* number of items in the queue */
+ int *nthreads; /* number of threads for each CPU */
+ int *nclients; /* number of processes for each CPU */
+ int *items; /* queue of CPUs to pick from */
+} cpu_generator_state;
+
+static cpu_generator_state cpu_generator_init(int ncpus);
+static void cpu_generator_refill(cpu_generator_state * state);
+static void cpu_generator_reset(cpu_generator_state * state);
+static int cpu_generator_thread(cpu_generator_state * state);
+static int cpu_generator_client(cpu_generator_state * state, int thread_cpu);
+static void cpu_generator_print(cpu_generator_state * state);
+static bool cpu_generator_check(cpu_generator_state * state);
+
+static void reset_pinning(TState *threads, int nthreads);
+
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
@@ -959,6 +1009,7 @@ usage(void)
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
" --show-script=NAME show builtin script code, then exit\n"
" --verbose-errors print messages of all errors\n"
+ " --pin-cpus MODE pin threads and backends to CPU (random or colocated)\n"
"\nCommon options:\n"
" --debug print debugging output\n"
" -d, --dbname=DBNAME database name to connect to\n"
@@ -6718,6 +6769,7 @@ main(int argc, char **argv)
{"verbose-errors", no_argument, NULL, 15},
{"exit-on-abort", no_argument, NULL, 16},
{"debug", no_argument, NULL, 17},
+ {"pin-cpus", required_argument, NULL, 18},
{NULL, 0, NULL, 0}
};
@@ -7071,6 +7123,15 @@ main(int argc, char **argv)
case 17: /* debug */
pg_logging_increase_verbosity();
break;
+ case 18: /* pin CPUs */
+ if (pg_strcasecmp(optarg, "random") == 0)
+ pinning_mode = CPU_PINNING_RANDOM;
+ else if (pg_strcasecmp(optarg, "colocated") == 0)
+ pinning_mode = CPU_PINNING_COLOCATED;
+ else
+ pg_fatal("invalid pinning method, expecting \"random\" or \"colocated\", got: \"%s\"",
+ optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -7368,6 +7429,96 @@ main(int argc, char **argv)
nclients_dealt += thread->nstate;
}
+ /* reset the CPU assignment, i.e. no pinning by default */
+ reset_pinning(threads, nthreads);
+
+ /* try to assign threads/clients to CPUs */
+ if (pinning_mode != CPU_PINNING_NONE)
+ {
+ int nprocs = get_nprocs();
+ cpu_generator_state state = cpu_generator_init(nprocs);
+
+retry:
+ /* start from scratch */
+ cpu_generator_reset(&state);
+
+ /* assign CPU to all threads */
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ thread->cpu = cpu_generator_thread(&state);
+ }
+
+ /*
+ * assign CPUs to backends, one at a time (for each thread)
+ *
+ * This helps to keep it balanced in the 'random' mode.
+ */
+ while (true)
+ {
+ /* did we find any unassigned backend? */
+ bool found = false;
+
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ /* skip backends with already assigned CPUs */
+ if (thread->state[j].cpu != -1)
+ continue;
+
+ if (pinning_mode == CPU_PINNING_RANDOM)
+ thread->state[j].cpu = cpu_generator_client(&state, thread->cpu);
+ else
+ {
+ state.nclients[thread->cpu]++;
+ thread->state[j].cpu = thread->cpu;
+ }
+
+ /* move to the next thread */
+ found = true;
+ break;
+ }
+ }
+
+ /* if we haven't found any unassigned backend, we're done */
+ if (!found)
+ break;
+ }
+
+ /* validate the assignments don't violate any of the restrictions */
+ if ((pinning_mode == CPU_PINNING_RANDOM) &&
+ !cpu_generator_check(&state))
+ {
+ reset_pinning(threads, nthreads);
+ goto retry;
+ }
+
+ /* print info about CPU assignments */
+ printf("============================\n");
+
+ cpu_generator_print(&state);
+
+ printf("----------------------------\n");
+
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ printf("thread %d CPU %d\n", i, thread->cpu);
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ printf(" client %d CPU %d\n", j, thread->state[j].cpu);
+ }
+ }
+
+ printf("============================\n");
+ }
+
/* all clients must be assigned to a thread */
Assert(nclients_dealt == nclients);
@@ -7519,6 +7670,39 @@ threadRun(void *arg)
}
}
+ /* pin the thread and backends for it's connections to the same CPU */
+ if (pinning_mode != CPU_PINNING_NONE)
+ {
+ cpu_set_t *cpusetp;
+ size_t cpusetsize;
+ size_t nprocs = get_nprocs();
+
+ cpusetp = CPU_ALLOC(nprocs);
+ cpusetsize = CPU_ALLOC_SIZE(nprocs);
+
+ CPU_ZERO_S(cpusetsize, cpusetp);
+ CPU_SET_S(thread->cpu, cpusetsize, cpusetp);
+
+ /* thread 0 dess not have an actual pthread */
+ if (thread->thread)
+ pthread_setaffinity_np(thread->thread, cpusetsize, cpusetp);
+ else
+ sched_setaffinity(getpid(), cpusetsize, cpusetp);
+
+ /* determine PID of the backend, pin it to the same CPU */
+ for (int i = 0; i < nstate; i++)
+ {
+ pid_t pid = PQbackendPID(state[i].con);
+
+ CPU_ZERO_S(cpusetsize, cpusetp);
+ CPU_SET_S(state[i].cpu, cpusetsize, cpusetp);
+
+ sched_setaffinity(pid, cpusetsize, cpusetp);
+ }
+
+ CPU_FREE(cpusetp);
+ }
+
/* GO */
THREAD_BARRIER_WAIT(&barrier);
@@ -7992,3 +8176,167 @@ socket_has_input(socket_set *sa, int fd, int idx)
}
#endif /* POLL_USING_SELECT */
+
+static cpu_generator_state
+cpu_generator_init(int ncpus)
+{
+ struct timeval tv;
+
+ cpu_generator_state state;
+
+ state.ncpus = ncpus;
+
+ state.nthreads = pg_malloc(sizeof(int) * ncpus);
+ state.nclients = pg_malloc(sizeof(int) * ncpus);
+
+ state.nitems = ncpus;
+ state.items = pg_malloc(sizeof(int) * ncpus);
+ for (int i = 0; i < ncpus; i++)
+ {
+ state.nthreads[i] = 0;
+ state.nclients[i] = 0;
+ state.items[i] = i;
+ }
+
+ gettimeofday(&tv, NULL);
+ srand48(tv.tv_usec);
+
+ return state;
+}
+
+static void
+cpu_generator_refill(cpu_generator_state * state)
+{
+ struct timeval tv;
+
+ state->items = pg_realloc(state->items,
+ (state->nitems + state->ncpus) * sizeof(int));
+
+ gettimeofday(&tv, NULL);
+
+ srand48(tv.tv_usec);
+
+ for (int i = 0; i < state->ncpus; i++)
+ state->items[state->nitems++] = i;
+}
+
+static void
+cpu_generator_reset(cpu_generator_state * state)
+{
+ state->nitems = 0;
+ cpu_generator_refill(state);
+
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ state->nthreads[i] = 0;
+ state->nclients[i] = 0;
+ }
+}
+
+static int
+cpu_generator_thread(cpu_generator_state * state)
+{
+ if (state->nitems == 0)
+ cpu_generator_refill(state);
+
+ while (true)
+ {
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
+
+ state->items[idx] = state->items[state->nitems - 1];
+ state->nitems--;
+
+ state->nthreads[cpu]++;
+
+ return cpu;
+ }
+}
+
+static int
+cpu_generator_client(cpu_generator_state * state, int thread_cpu)
+{
+ int min_clients;
+ bool has_valid_cpus = false;
+
+ for (int i = 0; i < state->nitems; i++)
+ {
+ if (state->items[i] != thread_cpu)
+ {
+ has_valid_cpus = true;
+ break;
+ }
+ }
+
+ if (!has_valid_cpus)
+ cpu_generator_refill(state);
+
+ min_clients = INT_MAX;
+ for (int i = 0; i < state->nitems; i++)
+ {
+ if (state->items[i] == thread_cpu)
+ continue;
+
+ min_clients = Min(min_clients, state->nclients[state->items[i]]);
+ }
+
+ while (true)
+ {
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
+
+ if (cpu == thread_cpu)
+ continue;
+
+ if (state->nclients[cpu] != min_clients)
+ continue;
+
+ state->items[idx] = state->items[state->nitems - 1];
+ state->nitems--;
+
+ state->nclients[cpu]++;
+
+ return cpu;
+ }
+}
+
+static void
+cpu_generator_print(cpu_generator_state * state)
+{
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ printf("CPU %d threads %d clients %d\n", i, state->nthreads[i], state->nclients[i]);
+ }
+}
+
+static bool
+cpu_generator_check(cpu_generator_state * state)
+{
+ int min_count = INT_MAX,
+ max_count = 0;
+
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ min_count = Min(min_count, state->nthreads[i] + state->nclients[i]);
+ max_count = Max(max_count, state->nthreads[i] + state->nclients[i]);
+ }
+
+ return (max_count - min_count <= 1);
+}
+
+static void
+reset_pinning(TState *threads, int nthreads)
+{
+ /* reset the CPU assignment, i.e. no pinning by default */
+ for (int i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ thread->cpu = -1;
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ thread->state[j].cpu = -1;
+ }
+ }
+}
--
2.50.1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-07 09:36 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
2 siblings, 0 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-08-07 09:36 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 8/7/25 11:24, Tomas Vondra wrote:
> Hi!
>
> Here's a slightly improved version of the patch series.
>
Ah, I made a mistake when generating the patches. The 0001 and 0002
patches are not part of the NUMA stuff, it's just something related to
benchmarking (addressing unrelated bottlenecks etc.). The actual NUMA
patches start with 0003.
Also, 0007, 0008 and 0009 should ultimately be collapsed into a single
patch. It's all about the clocksweep partitioning, I only kept those
separate to make it easier to see the changes and review.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-09 00:25 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
2 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-08-09 00:25 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
> 2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
> assumes each core / piece of RAM is assigned to a particular NUMA node.
There are systems in which some NUMA nodes do *not* contain any CPUs. E.g. if
you attach memory via a CXL/PCIe add-in card, rather than via the CPUs memory
controller. In that case numactl -H (and obviously also the libnuma APIs) will
report that the numa node is not associated with any CPU.
I don't currently have live access to such a system, but this PR piece happens
to have numactl -H output:
https://lenovopress.lenovo.com/lp2184-implementing-cxl-memory-on-linux-on-thinksystem-v4-servers
> numactl -H
> available: 4 nodes (0-3)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
> node 0 size: 1031904 MB
> node 0 free: 1025554 MB
> node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
> node 1 size: 1032105 MB
> node 1 free: 1024244 MB
> node 2 cpus:
> node 2 size: 262144 MB
> node 2 free: 262143 MB
> node 3 cpus:
> node 3 size: 262144 MB
> node 3 free: 262142 MB
> node distances:
> node 0 1 2 3
> 0: 10 21 14 24
> 1: 21 10 24 14
> 2: 14 24 10 26
> 3: 24 14 26 10
Note that node 2 & 3 don't have associated CPUs (and higher access costs).
I don't think this is common enough to worry about from a performance POV, but
we probably shouldn't crash if we encounter it...
> But it also cares about the cores (and the node for each core), because
> it uses that to pick the right partition for a backend. And here the
> situation is less clear, because the CPUs don't need to be assigned to a
> particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
>
> $ numactl --hardware
> available: 8 nodes (0-7)
> node 0 cpus: 0 1 2 3
> node 0 size: 992 MB
> node 0 free: 274 MB
> node 1 cpus: 0 1 2 3
> node 1 size: 1019 MB
> node 1 free: 327 MB
> ...
> node 0 1 2 3 4 5 6 7
> 0: 10 10 10 10 10 10 10 10
> 1: 10 10 10 10 10 10 10 10
> 2: 10 10 10 10 10 10 10 10
> 3: 10 10 10 10 10 10 10 10
> 4: 10 10 10 10 10 10 10 10
> 5: 10 10 10 10 10 10 10 10
> 6: 10 10 10 10 10 10 10 10
> 7: 10 10 10 10 10 10 10 10
> This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
> are not assigned to particular nodes - each core is mapped to all 8 NUMA
> nodes.
FWIW, you can get a different version of this with AMD Epyc too, if "L3 LLC as
NUMA" is enabled.
> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
I don't immediately see any libnuma functions that would care?
I also am somewhat curious about what getcpu() returns for the current node...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-12 11:04 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-08-12 11:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
On 8/9/25 02:25, Andres Freund wrote:
> Hi,
>
> On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
>> 2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
>> assumes each core / piece of RAM is assigned to a particular NUMA node.
>
> There are systems in which some NUMA nodes do *not* contain any CPUs. E.g. if
> you attach memory via a CXL/PCIe add-in card, rather than via the CPUs memory
> controller. In that case numactl -H (and obviously also the libnuma APIs) will
> report that the numa node is not associated with any CPU.
>
> I don't currently have live access to such a system, but this PR piece happens
> to have numactl -H output:
> https://lenovopress.lenovo.com/lp2184-implementing-cxl-memory-on-linux-on-thinksystem-v4-servers
>> numactl -H
>> available: 4 nodes (0-3)
>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
>> node 0 size: 1031904 MB
>> node 0 free: 1025554 MB
>> node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
>> node 1 size: 1032105 MB
>> node 1 free: 1024244 MB
>> node 2 cpus:
>> node 2 size: 262144 MB
>> node 2 free: 262143 MB
>> node 3 cpus:
>> node 3 size: 262144 MB
>> node 3 free: 262142 MB
>> node distances:
>> node 0 1 2 3
>> 0: 10 21 14 24
>> 1: 21 10 24 14
>> 2: 14 24 10 26
>> 3: 24 14 26 10
>
> Note that node 2 & 3 don't have associated CPUs (and higher access costs).
>
> I don't think this is common enough to worry about from a performance POV, but
> we probably shouldn't crash if we encounter it...
>
Right. I don't think the current patch would crash - I can't test it,
but I don't see why it would crash. In the worst case it'd end up with
partitions that are not ideal. The question is more what would an ideal
partitioning for buffers and PGPROC look like. Any opinions?
For PGPROC, it's simple - it doesn't make sense to allocate partitions
for nodes without CPUs.
For buffers, it probably does not really matter if a node does not have
any CPUs. If a node does not have any CPUs, that does not mean we should
not put any buffers on it. After all, CXL will never have any CPUs (at
least I think that's the case), and not using it for shared buffers
would be a bit strange. Although, it could still be used for page cache.
Maybe it should be "tiered" a bit more? The patch differentiates only
between partitions on "my" NUMA node vs. every other partition. Maybe it
should have more layers?
That'd make the "balancing" harder. But I wanted to make it a smarter
when handling cases with multiple partitions per NUMA node.
>
>> But it also cares about the cores (and the node for each core), because
>> it uses that to pick the right partition for a backend. And here the
>> situation is less clear, because the CPUs don't need to be assigned to a
>> particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
>>
>> $ numactl --hardware
>> available: 8 nodes (0-7)
>> node 0 cpus: 0 1 2 3
>> node 0 size: 992 MB
>> node 0 free: 274 MB
>> node 1 cpus: 0 1 2 3
>> node 1 size: 1019 MB
>> node 1 free: 327 MB
>> ...
>> node 0 1 2 3 4 5 6 7
>> 0: 10 10 10 10 10 10 10 10
>> 1: 10 10 10 10 10 10 10 10
>> 2: 10 10 10 10 10 10 10 10
>> 3: 10 10 10 10 10 10 10 10
>> 4: 10 10 10 10 10 10 10 10
>> 5: 10 10 10 10 10 10 10 10
>> 6: 10 10 10 10 10 10 10 10
>> 7: 10 10 10 10 10 10 10 10
>> This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
>> are not assigned to particular nodes - each core is mapped to all 8 NUMA
>> nodes.
>
> FWIW, you can get a different version of this with AMD Epyc too, if "L3 LLC as
> NUMA" is enabled.
>
>
>> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
>
> I don't immediately see any libnuma functions that would care?
>
Not sure what "care" means here. I don't think it's necessarily broken,
it's more about the APIs not making the situation very clear (or
convenient).
How do you determine nodes for a CPU, for example? The closest thing I
see is numa_node_of_cpu(), but that only returns a single node. Or how
would you determine the number of nodes with CPUs (so that we create
PGPROC partitions only for those)? I suppose that requires literally
walking all the nodes.
> I also am somewhat curious about what getcpu() returns for the current node...
>
It seems it only returns node 0. The cpu changes, but the node does not.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-12 14:24 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-08-12 14:24 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-08-12 13:04:07 +0200, Tomas Vondra wrote:
> Right. I don't think the current patch would crash - I can't test it,
> but I don't see why it would crash. In the worst case it'd end up with
> partitions that are not ideal. The question is more what would an ideal
> partitioning for buffers and PGPROC look like. Any opinions?
>
> For PGPROC, it's simple - it doesn't make sense to allocate partitions
> for nodes without CPUs.
>
> For buffers, it probably does not really matter if a node does not have
> any CPUs. If a node does not have any CPUs, that does not mean we should
> not put any buffers on it. After all, CXL will never have any CPUs (at
> least I think that's the case), and not using it for shared buffers
> would be a bit strange. Although, it could still be used for page cache.
For CXL memory to be really usable, I think we'd need nontrivial additional
work. CXL memory has considerably higher latency and lower throughput. You'd
*never* want things like BufferDescs or such on such nodes. And even the
buffered data itself, you'd want to make sure that frequently used data,
e.g. inner index pages, never end up on it.
Which leads to:
> Maybe it should be "tiered" a bit more?
Yes, for proper CXL support, we'd need a component that explicitly demotes and
promotes pages from "real" memory to CXL memory and the other way round. The
demotion is relatively easy, you'd probably just do it whenever you'd
otherwise throw out a victim buffer. When to promote back is harder...
> The patch differentiates only between partitions on "my" NUMA node vs. every
> other partition. Maybe it should have more layers?
Given the relative unavailability of CXL memory systems, I think just not
crashing is good enough for now...
> >> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
> >
> > I don't immediately see any libnuma functions that would care?
> >
>
> Not sure what "care" means here. I don't think it's necessarily broken,
> it's more about the APIs not making the situation very clear (or
> convenient).
What I mean is that I was looking through the libnuma functions and didn't see
any that would be affected by having multiple "local" NUMA nodes. But:
> How do you determine nodes for a CPU, for example? The closest thing I
> see is numa_node_of_cpu(), but that only returns a single node. Or how
> would you determine the number of nodes with CPUs (so that we create
> PGPROC partitions only for those)? I suppose that requires literally
> walking all the nodes.
I didn't think of numa_node_of_cpu().
As long as numa_node_of_cpu() returns *something* I think it may be good
enough. Nobody uses an RPi for high-throughput postgres workloads with a lot
of memory. Slightly sub-optimal mappings should really not matter.
I'm kinda wondering if we should deal with such fake numa systems by detecting
them and disabling our numa support.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-12 15:06 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-08-12 15:06 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
On 8/12/25 16:24, Andres Freund wrote:
> Hi,
>
> On 2025-08-12 13:04:07 +0200, Tomas Vondra wrote:
>> Right. I don't think the current patch would crash - I can't test it,
>> but I don't see why it would crash. In the worst case it'd end up with
>> partitions that are not ideal. The question is more what would an ideal
>> partitioning for buffers and PGPROC look like. Any opinions?
>>
>> For PGPROC, it's simple - it doesn't make sense to allocate partitions
>> for nodes without CPUs.
>>
>> For buffers, it probably does not really matter if a node does not have
>> any CPUs. If a node does not have any CPUs, that does not mean we should
>> not put any buffers on it. After all, CXL will never have any CPUs (at
>> least I think that's the case), and not using it for shared buffers
>> would be a bit strange. Although, it could still be used for page cache.
>
> For CXL memory to be really usable, I think we'd need nontrivial additional
> work. CXL memory has considerably higher latency and lower throughput. You'd
> *never* want things like BufferDescs or such on such nodes. And even the
> buffered data itself, you'd want to make sure that frequently used data,
> e.g. inner index pages, never end up on it.
>
OK, let's keep that out of scope for these patches and assume we're
dealing only with local memory. CXL could still be used by the OS for
page cache, of whatever.
What does that mean for the patch, though. Does it need a way to
configure which nodes to use? I argued to leave this to the OS/numactl,
and we'd just use whatever is made available to Postgres. But maybe
we'll need something within Postgres after all?
FWIW there's work needed a actually inherit NUMA info from the OS. Right
now the patches just use all NUMA nodes, indexed by 0 ... (N-1) etc. I
like the "registry" concept I used for buffer/PGPROC partitions, it made
the patches much simpler. Maybe we should use something like that for
NUMA info too. That is, at startup build a record of the NUMA layout,
and use this as source of truth everywhere (instead of using libnuma
from all those places).
> Which leads to:
>
>> Maybe it should be "tiered" a bit more?
>
> Yes, for proper CXL support, we'd need a component that explicitly demotes and
> promotes pages from "real" memory to CXL memory and the other way round. The
> demotion is relatively easy, you'd probably just do it whenever you'd
> otherwise throw out a victim buffer. When to promote back is harder...
>
Sounds very much like page cache (but that only works for buffered I/O).
>
>> The patch differentiates only between partitions on "my" NUMA node vs. every
>> other partition. Maybe it should have more layers?
>
> Given the relative unavailability of CXL memory systems, I think just not
> crashing is good enough for now...
>
The lowest of bars ;-)
>
>>>> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
>>>
>>> I don't immediately see any libnuma functions that would care?
>>>
>>
>> Not sure what "care" means here. I don't think it's necessarily broken,
>> it's more about the APIs not making the situation very clear (or
>> convenient).
>
> What I mean is that I was looking through the libnuma functions and didn't see
> any that would be affected by having multiple "local" NUMA nodes. But:
>
My question is a bit of a "reverse" to this. That is, how do we even
find (with libnuma) there are multiple local nodes?
>
>> How do you determine nodes for a CPU, for example? The closest thing I
>> see is numa_node_of_cpu(), but that only returns a single node. Or how
>> would you determine the number of nodes with CPUs (so that we create
>> PGPROC partitions only for those)? I suppose that requires literally
>> walking all the nodes.
>
> I didn't think of numa_node_of_cpu().
>
Yeah. I think most of the libnuma API is designed for each CPU belonging
to single NUMA node. I suppose we'd need to use numa_node_to_cpus() to
build this kind of information ourselves.
> As long as numa_node_of_cpu() returns *something* I think it may be good
> enough. Nobody uses an RPi for high-throughput postgres workloads with a lot
> of memory. Slightly sub-optimal mappings should really not matter.
>
I'm not really concerned about rpi, or the performance on it. I only use
it as an example of system with "weird" NUMA layout.
> I'm kinda wondering if we should deal with such fake numa systems by detecting
> them and disabling our numa support.
>
That'd be an option too, if we can identify such systems. We could do
that while building the "NUMA registry" I mentioned earlier.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-13 15:16 Andres Freund <[email protected]>
parent: Tomas Vondra <[email protected]>
2 siblings, 1 reply; 89+ messages in thread
From: Andres Freund @ 2025-08-13 15:16 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
> The patch does a much simpler thing - treat the weight as a "budget",
> i.e. number of buffers to allocate before proceeding to the "next"
> partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
> and then goes back to P1 in a round-robin way. The advantage is it can
> do away without a PRNG.
I think that's a good plan.
A few comments about the clock sweep patch:
- It'd be easier to review if BgBufferSync() weren't basically re-indented
wholesale. Maybe you could instead move the relevant code to a helper
function that's called by BgBufferSync() for each clock?
- I think choosing a clock sweep partition in every tick would likely show up
in workloads that do a lot of buffer replacement, particularly if buffers
in the workload often have a high usagecount (and thus more ticks are used).
Given that your balancing approach "sticks" with a partition for a while,
could we perhaps only choose the partition after exhausting that budget?
- I don't really understand what
> + /*
> + * Buffers that should have been allocated in this partition (but might
> + * have been redirected to keep allocations balanced).
> + */
> + pg_atomic_uint32 numRequestedAllocs;
> +
is intended for.
Adding yet another atomic increment for every clock sweep tick seems rather
expensive...
- I wonder if the balancing budgets being relatively low will be good
enough. It's not too hard to imagine that this frequent "partition choosing"
will be bad in buffer access heavy workloads. But it's probably the right
approach until we've measured it being a problem.
- It'd be interesting to do some very simple evaluation like a single
pg_prewarm() of a relation that's close to the size of shared buffers and
verify that we don't end up evicting newly read in buffers. I think your
approach should work, but verifying that...
I wonder if we could make some of this into tests somehow. It's pretty easy
to break this kind of thing and not notice, as everything just continues to
work, just a tad slower.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-08-13 16:36 Tomas Vondra <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-08-13 16:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
On 8/13/25 17:16, Andres Freund wrote:
> Hi,
>
> On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
>> The patch does a much simpler thing - treat the weight as a "budget",
>> i.e. number of buffers to allocate before proceeding to the "next"
>> partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
>> and then goes back to P1 in a round-robin way. The advantage is it can
>> do away without a PRNG.
>
> I think that's a good plan.
>
>
> A few comments about the clock sweep patch:
>
> - It'd be easier to review if BgBufferSync() weren't basically re-indented
> wholesale. Maybe you could instead move the relevant code to a helper
> function that's called by BgBufferSync() for each clock?
>
True, I'll rework it like that.
> - I think choosing a clock sweep partition in every tick would likely show up
> in workloads that do a lot of buffer replacement, particularly if buffers
> in the workload often have a high usagecount (and thus more ticks are used).
> Given that your balancing approach "sticks" with a partition for a while,
> could we perhaps only choose the partition after exhausting that budget?
>
That should be possible, yes. By "exhausting budget" you mean going
through all the partitions, right?
> - I don't really understand what
>
>> + /*
>> + * Buffers that should have been allocated in this partition (but might
>> + * have been redirected to keep allocations balanced).
>> + */
>> + pg_atomic_uint32 numRequestedAllocs;
>> +
>
> is intended for.
>
> Adding yet another atomic increment for every clock sweep tick seems rather
> expensive...
>
For the balancing (to calculate the budgets), we need to know the number
of allocation requests for each partition, before some of the requests
got redirected to other partitions. We can't use the number of "actual"
allocations. But it seems useful to have both - one to calculate the
budgets, the other to monitor how balanced the result is.
I haven't seen the extra atomic in profiles, even on workloads that do a
lot of buffer allocations (e.g. seqscan with datasets > shared buffers).
But if that happens, I think there are ways to mitigate that.
>
> - I wonder if the balancing budgets being relatively low will be good
> enough. It's not too hard to imagine that this frequent "partition choosing"
> will be bad in buffer access heavy workloads. But it's probably the right
> approach until we've measured it being a problem.
>
I don't follow. How would making the budgets higher change any of this?
Anyway, I think choosing the partitions less frequently - e.g. only
after consuming budget for the current partition, or going "full cycle",
would make this a non-issue.
>
> - It'd be interesting to do some very simple evaluation like a single
> pg_prewarm() of a relation that's close to the size of shared buffers and
> verify that we don't end up evicting newly read in buffers. I think your
> approach should work, but verifying that...
>
Will try.
> I wonder if we could make some of this into tests somehow. It's pretty easy
> to break this kind of thing and not notice, as everything just continues to
> work, just a tad slower.
>
Do you mean a test that'd be a part of make check, or a standalone test?
AFAICS any meaningful test would need to be fairly expensive, so
probably not a good fit for make check.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-09-11 08:32 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 2 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-09-11 08:32 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Here's a fresh version of the NUMA patch series. There's a number of
substantial improvements:
1) Rebase to current master, particularly on top of 2c7894052759 which
removed the freelist. The patch that partitioned the freelist is gone.
2) A bunch of fixes, and it now passes CI workflows on github, and all
other testing I did. Of course, more testing is needed.
3) It builds with/without libnuma support, and so on.
4) The separate GUCs were replaced by a single list GUC, similar to what
we do for debug_io_direct. The GUC is called debug_numa, and accepts
"buffets" and "procs", to partition the two areas.
I'm considering adding a "clock-sweep" option in a future patch. I
didn't do that here, because the buffer partitioning is already enabled
by "buffers", and the clock-sweep just builds on that (partitioning the
same way, pretty much).
5) It also works with EXEC_BACKEND, but this turned out to be a bit more
challenging than expected. The trouble is some of the parameters (e.g.
memory page size) are used both in the "size" and "init" phases, and we
need to be extra careful to not make something "inconsistent".
For example, we may get confused about the memory page size. The "size"
happens before allocation, and at that point we don't know if we succeed
in getting enough huge pages. When "init" happens, we already know that,
so our "memory page size" could be different. We must be careful, e.g.
to not need more memory than we requested.
This is a general problem, but the EXEC_BACKEND makes it a bit trickier.
In regular fork() case we can simply set some global variables in "size"
and use them later in "init", but that doesn't work for EXEC_BACKEND.
The current approach simply does the calculations twice, in a way that
should end with the same results. But I'm not 100% happy with it, and I
suspect it just confirms we should store the results in shmem memory
(which is what I called "NUMA registry" before). That's still a TODO.
6) BufferDescPadded is 64B everywhere. Originally, the padding was
applied only on 64-bit platforms, and on 32-bit systems the struct was
left at 60B. But that is not compatible with buffer partitioning, which
relies on the memory page being a multiple of BufferDescPadded.
Perhaps it could be relaxed (so that a BufferDesc might span two memory
pages), but this seems like the cleanes solution. I don't expect it to
make any measurable difference.
7) I've moved some of the code to BgBufferSyncPartition, to make review
easier (without all the indentation changes).
8) I've realized some of the TAP tests occasionally fail with
ERROR: no unpinned buffers
and I think I know why. Some of the tests set shared_buffers to a very
low value - like 1MB or even 128kB, and StrategyGetBuffer() may search
only a single partition (but not always). We may run out of unpinned
buffers in that one partition.
This apparently happens more easily on rpi5, due to the weird NUMA
layout (there are 8 nodes with memory, but getcpu() reports node 0 for
all cores).
I suspect the correct fix is to ensure StrategyGetBuffer() scans all
partitions, if there are no unpinned buffers in the current one. On
realistic setups this shouldn't happen very often, I think.
The other issue I just realized is that StrategyGetBuffer() recalculates
the partition index over and over, which seems unnecessary (and possibly
expensive, due to the modulo). And it also does too many loops, because
it used NBuffers instead of the partition size. I'll fix those later.
9) I'm keeping the cloc-sweep balancing patches separate for now. In the
end all the clock-sweep patches should be merged, but it keeps the
changes easier to review this way, I think.
10) There's not many changes in the PGPROC partitioning patch. I merely
fixed issues that broke it on EXEC_BACKEND, and did some smaller tweaks.
11) I'm not including the experimental patches to pin backends to CPUs
(or nodes), and so on. It's clear those are unlikely to go in, so it'd
be just a distraction.
12) What's the right / portable way to determine the current CPU for a
process? The clock-sweep / PGPROC patches need this to pick the right
partition, but it's not clear to me which API is the most portable. In
the end I used sched_getcpu(), and then numa_node_of_cpu(), but maybe
there's a better way.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20250911-0001-NUMA-shared-buffers-partitioning.patch (41.7K, ../../[email protected]/2-v20250911-0001-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 318f2441cc67daf7fb2ba672900a2bd4dfdea170 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 28 Jul 2025 14:01:37 +0200
Subject: [PATCH v20250911 1/8] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
This allows partitioning clock-sweep in a later patch, with one clock
hand per partition.
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 26 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 627 +++++++++++++++++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 23 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 859 insertions(+), 16 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..fb9003c011e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,26 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 3df04c98959..8a0a4bd5cd6 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -777,3 +779,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..f51c7db7855 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +35,23 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
/*
* Data Structures:
@@ -70,19 +98,87 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
- /* Align descriptors to a cacheline boundary. */
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
+
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +208,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -148,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -175,5 +284,505 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
+}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 0da01627cfe..d18666a94cb 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -899,6 +899,16 @@
boot_val => 'true',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'sync_replication_slots', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Enables a physical standby to synchronize logical failover replication slots from the primary server.',
variable => 'sync_replication_slots',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 00c8376cf4d..d17ee9ca861 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index dfd614f7ca4..294188e21c5 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -275,10 +275,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -288,7 +288,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -321,6 +321,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -480,4 +481,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..e52fca9e483 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -143,6 +143,28 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int numa_node; /* NUMA node (-1 no node) */
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* forward declared, to avoid having to expose buf_internals.h here */
struct WritebackContext;
@@ -319,6 +341,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 3368a43a338..8ee0e7d211c 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -106,6 +109,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -128,4 +161,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a13e8162890..50195718294 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.0
[text/x-patch] v20250911-0002-NUMA-clockweep-partitioning.patch (35.5K, ../../[email protected]/3-v20250911-0002-NUMA-clockweep-partitioning.patch)
download | inline diff:
From 38ccd93b4a70d65e16da3303e9f94851c3a3fb5a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v20250911 2/8] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 11 +
src/backend/storage/buffer/bufmgr.c | 186 +++++----
src/backend/storage/buffer/freelist.c | 353 ++++++++++++++++--
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 503 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index fb9003c011e..6676e807034 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -16,7 +16,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8a0a4bd5cd6..c9dfc8a1b82 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -839,12 +847,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -860,6 +878,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index f51c7db7855..dd9f51529b4 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -716,6 +716,17 @@ BufferPartitionGet(int idx, int *node, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ if (num_partitions)
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
/* XXX the GUC hooks should probably be somewhere else? */
bool
check_debug_numa(char **newval, void **extra, GucSource source)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index fe470de63f2..121134bb94c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3580,33 +3580,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3634,25 +3630,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3664,17 +3651,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3682,11 +3669,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3706,9 +3693,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3722,15 +3709,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3738,9 +3726,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3750,7 +3738,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3758,10 +3746,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3788,7 +3776,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3813,20 +3801,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3840,7 +3828,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3870,8 +3858,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7d59a92bd1a..d5f8f28f562 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,34 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +132,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +144,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +152,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +185,117 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
+ */
+static int
+calculate_partition_index(void)
+{
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -222,9 +365,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -269,6 +438,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -276,37 +485,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -343,6 +559,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -350,6 +569,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -365,6 +588,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -382,7 +617,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -394,15 +630,44 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
@@ -739,3 +1004,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 294188e21c5..5cce690933b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -439,7 +439,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -485,5 +487,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e52fca9e483..9ade69e53b5 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -355,6 +355,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 50195718294..b68f75b7f31 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -427,6 +427,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.0
[text/x-patch] v20250911-0003-NUMA-clocksweep-allocation-balancing.patch (25.3K, ../../[email protected]/4-v20250911-0003-NUMA-clocksweep-allocation-balancing.patch)
download | inline diff:
From 6a4444f319394ac88abe27faf4ccba22ea1a0dbe Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 10 Sep 2025 18:56:28 +0200
Subject: [PATCH v20250911 3/8] NUMA: clocksweep allocation balancing
If backends only allocate buffers from the "local" partition, this could
cause significant misbalance - some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 6676e807034..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -22,7 +22,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index c9dfc8a1b82..f6831f60b9e 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -795,6 +797,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -826,6 +830,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -847,11 +857,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -860,8 +876,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -890,6 +914,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 121134bb94c..8315105394d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3884,6 +3884,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d5f8f28f562..349b626db3b 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -132,7 +168,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -144,7 +206,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -291,11 +353,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -365,7 +475,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -438,6 +548,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -464,6 +792,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -658,7 +987,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1007,8 +1350,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1016,11 +1361,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5cce690933b..1b4dae180e0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -439,6 +439,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 9ade69e53b5..fb4162eb764 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -356,11 +356,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.0
[text/x-patch] v20250911-0004-NUMA-weighted-clocksweep-balancing.patch (5.1K, ../../[email protected]/5-v20250911-0004-NUMA-weighted-clocksweep-balancing.patch)
download | inline diff:
From 391a25e8cbe88e9a018a5123b3d40ffec607891e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20250911 4/8] NUMA: weighted clocksweep balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 349b626db3b..5cf9a565914 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -605,6 +605,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -631,16 +645,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -648,8 +673,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -712,6 +743,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -719,7 +754,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -733,22 +768,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.0
[text/x-patch] v20250911-0005-NUMA-partition-PGPROC.patch (48.7K, ../../[email protected]/6-v20250911-0005-NUMA-partition-PGPROC.patch)
download | inline diff:
From 4cdfff63d1a903902ce182b7e723ff6d2d09a84e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 8 Sep 2025 13:11:02 +0200
Subject: [PATCH v20250911 5/8] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 ++++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 +--
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 532 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 705 insertions(+), 70 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index f6831f60b9e..a859962f5f8 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -932,3 +935,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index d8e2fce2c99..7745a197470 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index a38979c50e4..01356365eaa 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -106,8 +106,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..011fecfc58b 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index dd9f51529b4..2fd7f937ffb 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -766,6 +766,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 5cf9a565914..a6657c0fc13 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -467,7 +467,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 4cc7f645c31..d01f486876d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..08d1900fb59 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,32 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -76,8 +87,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -90,6 +101,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +134,36 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
return size;
}
@@ -129,6 +188,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -140,12 +253,16 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
return size;
}
@@ -191,7 +308,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -210,6 +327,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -224,6 +344,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,19 +370,104 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +500,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,9 +648,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -435,7 +714,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -646,7 +969,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1049,7 +1372,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1098,7 +1421,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1988,7 +2311,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2386,168 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b68f75b7f31..6a970998e5c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1876,6 +1876,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-09-11 15:41 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-09-11 15:41 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
On 9/11/25 10:32, Tomas Vondra wrote:
> ...
>
> For example, we may get confused about the memory page size. The "size"
> happens before allocation, and at that point we don't know if we succeed
> in getting enough huge pages. When "init" happens, we already know that,
> so our "memory page size" could be different. We must be careful, e.g.
> to not need more memory than we requested.
I forgot to mention the other issue with huge pages on NUMA. I already
reported [1] it's trivial to crash with a SIGBUS, because
(1) huge pages get reserved on all NUMA nodes (evenly)
(2) the decision whether to use huge pages is done by mmap(), which only
needs to check if there are enough huge pages in total
(3) numa_tonode_memory is called later, and does not verify if the
target node has enough free pages (I'm not sure it should / can)
(4) we only partition (and locate to NUMA nodes) some of the memory, and
the rest (which is much smaller, but still sizeable) is likely causing
"imbalance" - it gets placed on one (random) node, and it then does not
have enough space for the stuff we explicitly placed there
(5) then at some point we try accessing one of the shared buffers, that
triggers page fault, tries to get a huge page on the NUMA node, realizes
there are no free huge pages, and crashes with SIGBUS
It clearly is not an option to just let it crash, but I still don't have
a great idea how to address it. The only idea I have is to manually
interleave the whole shared memory (when using huge pages), page by
page, so that this imbalance does not happen.
But it's harder than it looks, because we don't necessarily partition
everything evenly. For example, one node can get a smaller chunk of
shared buffers, because we try to partition buffers and buffers
descriptors in a "nice" way. The PGPROC stuff is also not distributed
quite evenly (e.g. aux/2pc entries are not mapped to any node).
A different approach would be to calculate how many per-node huge pages
we'll need (for the stuff we partition explicitly - buffers and PGPROC),
and then the rest of the memory that can get placed on any node. And
require the "maximum" number of pages that can get placed on any node.
But that's annoying wasteful, because every other node will end up with
unusable memory.
regards
[1]
https://www.postgresql.org/message-id/71a46484-053c-4b81-ba32-ddac050a8b5d%40vondra.me
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-09-18 21:04 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-09-18 21:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; PostgreSQL Hackers <[email protected]>
On 9/11/25 10:32, Tomas Vondra wrote:
> ...
>
> 8) I've realized some of the TAP tests occasionally fail with
>
> ERROR: no unpinned buffers
>
> and I think I know why. Some of the tests set shared_buffers to a very
> low value - like 1MB or even 128kB, and StrategyGetBuffer() may search
> only a single partition (but not always). We may run out of unpinned
> buffers in that one partition.
>
> This apparently happens more easily on rpi5, due to the weird NUMA
> layout (there are 8 nodes with memory, but getcpu() reports node 0 for
> all cores).
>
> I suspect the correct fix is to ensure StrategyGetBuffer() scans all
> partitions, if there are no unpinned buffers in the current one. On
> realistic setups this shouldn't happen very often, I think.
>
> The other issue I just realized is that StrategyGetBuffer() recalculates
> the partition index over and over, which seems unnecessary (and possibly
> expensive, due to the modulo). And it also does too many loops, because
> it used NBuffers instead of the partition size. I'll fix those later.
Here's a version fixing this issue (in the 0006 part). It modifies
StrategyGetBuffer() to walk through all the partitions, in a round-robin
manner. The way it steps to the next partition is a bit ugly, but it
works and I'll think about some better way.
I haven't done anything about the other issue (the one with huge pages
reserved on NUMA nodes, and SIGBUS).
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20250918-0001-NUMA-shared-buffers-partitioning.patch (41.7K, ../../[email protected]/2-v20250918-0001-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 9141bb89bce485873978e1ae3988b43e675ee047 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20250918 1/6] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
This allows partitioning clock-sweep in a later patch, with one clock
hand per partition.
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 26 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 627 +++++++++++++++++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 23 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 859 insertions(+), 16 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..fb9003c011e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,26 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 3df04c98959..8a0a4bd5cd6 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -777,3 +779,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..f51c7db7855 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +35,23 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
/*
* Data Structures:
@@ -70,19 +98,87 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
- /* Align descriptors to a cacheline boundary. */
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
+
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +208,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -148,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -175,5 +284,505 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
+}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 6bc6be13d2a..65bef2b3c1b 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -899,6 +899,16 @@
boot_val => 'true',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'sync_replication_slots', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Enables a physical standby to synchronize logical failover replication slots from the primary server.',
variable => 'sync_replication_slots',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 00c8376cf4d..d17ee9ca861 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index dfd614f7ca4..294188e21c5 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -275,10 +275,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -288,7 +288,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -321,6 +321,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -480,4 +481,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 47360a3d3d8..1618caf1c2c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -146,6 +146,28 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int numa_node; /* NUMA node (-1 no node) */
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
@@ -319,6 +341,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 3368a43a338..8ee0e7d211c 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -106,6 +109,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -128,4 +161,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e90af5b2ad3..b3f0504008e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.0
[text/x-patch] v20250918-0002-NUMA-clockweep-partitioning.patch (35.5K, ../../[email protected]/3-v20250918-0002-NUMA-clockweep-partitioning.patch)
download | inline diff:
From e7145b7e771487012aa0c723309353723db93172 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v20250918 2/6] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 11 +
src/backend/storage/buffer/bufmgr.c | 186 +++++----
src/backend/storage/buffer/freelist.c | 353 ++++++++++++++++--
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 503 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index fb9003c011e..6676e807034 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -16,7 +16,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8a0a4bd5cd6..c9dfc8a1b82 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -839,12 +847,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -860,6 +878,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index f51c7db7855..dd9f51529b4 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -716,6 +716,17 @@ BufferPartitionGet(int idx, int *node, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ if (num_partitions)
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
/* XXX the GUC hooks should probably be somewhere else? */
bool
check_debug_numa(char **newval, void **extra, GucSource source)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index fe470de63f2..121134bb94c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3580,33 +3580,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3634,25 +3630,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3664,17 +3651,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3682,11 +3669,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3706,9 +3693,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3722,15 +3709,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3738,9 +3726,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3750,7 +3738,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3758,10 +3746,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3788,7 +3776,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3813,20 +3801,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3840,7 +3828,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3870,8 +3858,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7d59a92bd1a..d5f8f28f562 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,34 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +132,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +144,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +152,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +185,117 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
+ */
+static int
+calculate_partition_index(void)
+{
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -222,9 +365,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -269,6 +438,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -276,37 +485,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -343,6 +559,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -350,6 +569,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -365,6 +588,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -382,7 +617,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -394,15 +630,44 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
@@ -739,3 +1004,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 294188e21c5..5cce690933b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -439,7 +439,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -485,5 +487,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 1618caf1c2c..7d66e8276cd 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -355,6 +355,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b3f0504008e..2b7ce2d0b9f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -427,6 +427,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.0
[text/x-patch] v20250918-0003-NUMA-clocksweep-allocation-balancing.patch (25.3K, ../../[email protected]/4-v20250918-0003-NUMA-clocksweep-allocation-balancing.patch)
download | inline diff:
From 1707ae15be8f0311219c4ba84b15a41d9bde6c69 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 10 Sep 2025 18:56:28 +0200
Subject: [PATCH v20250918 3/6] NUMA: clocksweep allocation balancing
If backends only allocate buffers from the "local" partition, this could
cause significant misbalance - some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 6676e807034..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -22,7 +22,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index c9dfc8a1b82..f6831f60b9e 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -795,6 +797,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -826,6 +830,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -847,11 +857,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -860,8 +876,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -890,6 +914,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 121134bb94c..8315105394d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3884,6 +3884,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d5f8f28f562..349b626db3b 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -132,7 +168,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -144,7 +206,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -291,11 +353,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -365,7 +475,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -438,6 +548,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -464,6 +792,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -658,7 +987,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1007,8 +1350,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1016,11 +1361,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5cce690933b..1b4dae180e0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -439,6 +439,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7d66e8276cd..1ca0d4c375e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -356,11 +356,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.0
[text/x-patch] v20250918-0004-NUMA-weighted-clocksweep-balancing.patch (5.1K, ../../[email protected]/5-v20250918-0004-NUMA-weighted-clocksweep-balancing.patch)
download | inline diff:
From 5631cd15f07c368153014464c4ce58ab0838a317 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20250918 4/6] NUMA: weighted clocksweep balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 349b626db3b..5cf9a565914 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -605,6 +605,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -631,16 +645,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -648,8 +673,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -712,6 +743,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -719,7 +754,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -733,22 +768,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.0
[text/x-patch] v20250918-0005-NUMA-partition-PGPROC.patch (48.7K, ../../[email protected]/6-v20250918-0005-NUMA-partition-PGPROC.patch)
download | inline diff:
From 5247bf1c147ec9593e72168fec3579910b4f262c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 8 Sep 2025 13:11:02 +0200
Subject: [PATCH v20250918 5/6] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 ++++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 +--
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 532 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 705 insertions(+), 70 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index f6831f60b9e..a859962f5f8 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -932,3 +935,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index d8e2fce2c99..7745a197470 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 79708e59259..92db67fa5f9 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..011fecfc58b 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index dd9f51529b4..2fd7f937ffb 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -766,6 +766,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 5cf9a565914..a6657c0fc13 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -467,7 +467,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 4cc7f645c31..d01f486876d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 96f29aafc39..70ccfebef55 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,32 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -76,8 +87,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -90,6 +101,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +134,36 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
return size;
}
@@ -129,6 +188,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -140,12 +253,16 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
return size;
}
@@ -191,7 +308,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -210,6 +327,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -224,6 +344,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,19 +370,104 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +500,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,9 +648,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -435,7 +714,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -646,7 +969,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1049,7 +1372,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1098,7 +1421,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1988,7 +2311,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2386,168 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2b7ce2d0b9f..912dd4ac100 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1876,6 +1876,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.0
[text/x-patch] v20250918-0006-fixup-StrategyGetBuffer.patch (6.6K, ../../[email protected]/7-v20250918-0006-fixup-StrategyGetBuffer.patch)
download | inline diff:
From e78f37f4769225522ec4ce828fbc3a3e46bcdf6e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:42:34 +0200
Subject: [PATCH v20250918 6/6] fixup: StrategyGetBuffer
StrategyGetBuffer is expected to scan all clock-sweep partitions, not
just the "local" one. So start at the optimal one (as calculated by
ChooseClockSweep), and then advance to the next one in a round-robin
way, until we find a free / unpinned buffer.
---
src/backend/storage/buffer/freelist.c | 94 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 65 insertions(+), 34 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index a6657c0fc13..d911986d7ce 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -169,6 +169,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -203,10 +206,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -425,8 +427,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
- uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -480,34 +482,68 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
- *
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+
for (;;)
{
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+ uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+
+ trycounter = sweep->numBuffers;
+ for (;;)
+ {
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot use
@@ -521,7 +557,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
local_buf_state -= BUF_USAGECOUNT_ONE;
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
}
else
{
@@ -542,7 +578,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* infinite loop.
*/
UnlockBufHdr(buf, local_buf_state);
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
UnlockBufHdr(buf, local_buf_state);
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-12 23:58 Alexey Makhmutov <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Alexey Makhmutov @ 2025-10-12 23:58 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
Hi Tomas,
Thank you very much for working on this problem and the entire line of
patches prepared! I've tried to play with these patches a little and
here are some my observations and suggestions.
In the current implementation we try to use all available NUMA nodes on
the machine, however it's often useful to limit the database only to a
set of specific nodes, so that other nodes can be used for other
processes. In my testing I was trying to use one node out of four for
the client program, so I'd liked to limit the database to the remaining
nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
start the cluster, so the obvious choice for me was to use the
'numa_get_membind' function instead of 'numa_num_configured_nodes' to
get the list of usable nodes. However, it is much easier to work with
logical nodes in the [0; n] range inside the PG code, so I've decided to
add mapping between 'logical nodes' (0-n in PG) to a set of physical
nodes actually returned by 'numa_get_membind'. We may need to map number
in both directions, so two translation tables are allocated and filled
at the first usage of 'pg_numa' functions. It also seems to be a good
idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
'numa_...' calls in it and this also allows us to hide this mapping in
static functions. Here is the patch, which I've used to test this idea:
https://github.com/Lerm/postgres/commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161.
This idea probably could be extended by adding some view to expose this
mapping to the user (at least for testing purposes) and allow to
explicitly override this mapping with a GUC setting. With such GUC
setting we would be able to control PG memory usage on NUMA nodes
without the need for systemd resource control or numactl parameters.
Next, I've noticed some problems related to the size alignment for
'numa_tonode_memory' call in 'pg_numa_move_to_node' function. The
documentation for the 'numa_tonode_memory' says that 'The size
argument will be rounded up to a multiple of the system page size'.
However this does not work well with huge pages as alignment is
performed for the default kernel page size (i.e. 4K in most cases). If
addr + size value (rounded to the default page size) does not cover the
entire huge page, then such invocation seems to be processed incorrectly
and allocation policy is not applied for next pages access in such
segment. At least this was the behavior I've observed on Debian 12 /
6.1.40 kernel (i.e. '/proc/<pid>/numa_maps' shows that the segment
contains pages from wrong nodes).
There are two location at which we could face such situation in current
patches. First is related to buffers partitions mapping. With current
code we basically ensure that combined size of all partitions for a
single node is aligned to (huge) page size (as size is bound to the
number of descriptors on one page), but individual partition is not
explicitly aligned to this size. So, we could get the situation in which
single page is split between adjacent partitions (e.g. 32GB buffers
split by 3 nodes). With current code we will try to map each partition
independently, which will results in unaligned calls to
'numa_tonode_memory', so resulting mapping will be incorrect. We could
either try to choose size for individual partition to align it to the
desired page size or map all the partitions for a single node using a
single 'pg_numa_move_to_node' invocation. During testing I've used the
second approach, so here is the change to implement such logic:
https://github.com/Lerm/postgres/commit/ee8b3603afd6d89e67b755dadc8e4c25ffba88be.
The second location which could expose the same problem is related to
the mapping of PGPROC arrays in 'pgproc_partition_init': here we need to
align pointer to the end PGPROC partition. There seems to be also two
additional problems with PGPROC partitioning: we need to account
additional padding pages in 'PGProcShmemSize' (using the same logic as
with fplocks) and we should not call 'MemSet(ptr, 0, ...)' prior to
partitions mapping call (otherwise it will be mapped to current node).
Here is a potential change, which tries to address these problems:
https://github.com/Lerm/postgres/commit/eaf12776f59ff150735d0f187595fc8ce3f0a872.
There are also some potential problems with buffers distribution between
nodes. I have a feeling that current logic in
'buffer_partitions_prepare' does not work correctly if number of buffers
is enough to cover just a single partition per node, but total number of
nodes is below MIN_BUFFER_PARTITIONS (i.e. 2 or 3). In this case we will
set 'numa_can_partition' to 'true', but will allocate 2 partitions per
node (so, 4 or 6 partitions in total), while we can fill just 2 or 3
partition and leaving remaining partitions empty. This should violate
the last assert check, as last partition will get zero buffers in this
case. Another issue is related to the usage of 1GB pages, as minimal
size for buffers partitions is limited by the minimal number of buffer
descriptors in a single page. For 2MB pages this gives 2097152 / 64 * 8K
= 256M as minimal size for partition, but for 1GB page the minimal size
is equal to 1GB / 64 * 8K = 128GB. So, if we assume 4 as minimal number
of partitions, then for 2MB pages we need just 1GB for shared_buffers to
enable partitioning (which seems a perfectly fine minimal limit for most
cases), but for 1GB pages we need to allocate at least 512GB to allow
buffers partitioning. Certainly, 1GB pages are usually used on large
machines with large number of buffers allocated, but still it may be
useful to allow configurations with 32GB or 64GB buffer cache to use
both 1GB pages and buffers partitioning at the same time. However, I
don't see an easy way to achieve this with the current logic. We either
need to allow usage of different page sizes here (i.e. 2MB for
descriptors and 1GB for buffers) or combine both buffers and its
descriptors in a single object (i.e. 'buffer chunk', which cover enough
buffers and their descriptors to fit into one or several memory pages),
effectively replacing both buffers and descriptors arrays with an array
of such 'chunks'. The latter solution may also help with dynamic buffer
cache resizing (as we may just add additional 'chunks' in this case) and
also increase TLB-hits with 1GB page (as both descriptor and its buffer
will be likely located in the same page). However, both these changes
seems to be quite large.
I've tried also to run some benchmarks on my server: I've got some
improvements in 'pgbench/tpcb-like'results - about 8%, but only with
backends pinning to NUMA node (i.e. adjusting your previous pinning
patch to 'debug_numa' GUC:
https://github.com/Lerm/postgres/commit/5942a3e12c7c501aa9febb63972a039e7ce00c20).
For 'select-only' scenario the gain is more substantial (about 15%), but
these tests are tricky, as they are more sensitive to other server
settings and specific functions layout in compiled code, so they need
more checks.
Thank you again for sharing these patches!
Thanks,
Alexey
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-13 11:09 Tomas Vondra <[email protected]>
parent: Alexey Makhmutov <[email protected]>
0 siblings, 2 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-10-13 11:09 UTC (permalink / raw)
To: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
On 10/13/25 01:58, Alexey Makhmutov wrote:
> Hi Tomas,
>
> Thank you very much for working on this problem and the entire line of
> patches prepared! I've tried to play with these patches a little and
> here are some my observations and suggestions.
>
> In the current implementation we try to use all available NUMA nodes on
> the machine, however it's often useful to limit the database only to a
> set of specific nodes, so that other nodes can be used for other
> processes. In my testing I was trying to use one node out of four for
> the client program, so I'd liked to limit the database to the remaining
> nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
> start the cluster, so the obvious choice for me was to use the
> 'numa_get_membind' function instead of 'numa_num_configured_nodes' to
> get the list of usable nodes. However, it is much easier to work with
> logical nodes in the [0; n] range inside the PG code, so I've decided to
> add mapping between 'logical nodes' (0-n in PG) to a set of physical
> nodes actually returned by 'numa_get_membind'. We may need to map number
> in both directions, so two translation tables are allocated and filled
> at the first usage of 'pg_numa' functions. It also seems to be a good
> idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
> 'numa_...' calls in it and this also allows us to hide this mapping in
> static functions. Here is the patch, which I've used to test this idea:
> https://github.com/Lerm/postgres/
> commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. This idea probably
> could be extended by adding some view to expose this mapping to the user
> (at least for testing purposes) and allow to explicitly override this
> mapping with a GUC setting. With such GUC setting we would be able to
> control PG memory usage on NUMA nodes without the need for systemd
> resource control or numactl parameters.
>
I've argued to keep this out of scope for v1, to keep it smaller and
simpler. I'm not against adding that feature, though. If someone writes
a patch to support this. I suppose the commit you linked is a step in
that direction.
I agree we should isolate libnuma calls to pg_numa.{c,h}. I wasn't quite
consistent when doing that.
> Next, I've noticed some problems related to the size alignment for
> 'numa_tonode_memory' call in 'pg_numa_move_to_node' function. The
> documentation for the 'numa_tonode_memory' says that 'The size
> argument will be rounded up to a multiple of the system page size'.
> However this does not work well with huge pages as alignment is
> performed for the default kernel page size (i.e. 4K in most cases). If
> addr + size value (rounded to the default page size) does not cover the
> entire huge page, then such invocation seems to be processed incorrectly
> and allocation policy is not applied for next pages access in such
> segment. At least this was the behavior I've observed on Debian 12 /
> 6.1.40 kernel (i.e. '/proc/<pid>/numa_maps' shows that the segment
> contains pages from wrong nodes).
>
I'm not sure I understand. Are you suggesting there's a bug in the
patch, the kernel, or somewhere else?
There's definitely a possibility of confusion with huge pages, no doubt
about that. The default "system page size" is 4KB, but we need to
process whole huge pages.
But this is exactly why (with hugepages) the code aligns everything to
huge page boundary, and sizes everything as a multiple of huge page. At
least I think so. Maybe I remember wrong?
> There are two location at which we could face such situation in current
> patches. First is related to buffers partitions mapping. With current
> code we basically ensure that combined size of all partitions for a
> single node is aligned to (huge) page size (as size is bound to the
> number of descriptors on one page), but individual partition is not
> explicitly aligned to this size. So, we could get the situation in which
> single page is split between adjacent partitions (e.g. 32GB buffers
> split by 3 nodes). With current code we will try to map each partition
> independently, which will results in unaligned calls to
> 'numa_tonode_memory', so resulting mapping will be incorrect. We could
> either try to choose size for individual partition to align it to the
> desired page size or map all the partitions for a single node using a
> single 'pg_numa_move_to_node' invocation. During testing I've used the
> second approach, so here is the change to implement such logic: https://
> github.com/Lerm/postgres/commit/ee8b3603afd6d89e67b755dadc8e4c25ffba88be.
>
Can you actually demonstrate this? The code does these two things:
* calculate min_node_buffers so that buffers/descriptors are a multiple
of page size (either 4K or huge page)
* align buffers and descriptors to memory page
TYPEALIGN(buffer_align, ...)
I believe this is sufficient to ensure nothing gets split / mapped
incorrectly. Maybe this fails sometimes?
> The second location which could expose the same problem is related to
> the mapping of PGPROC arrays in 'pgproc_partition_init': here we need to
> align pointer to the end PGPROC partition. There seems to be also two
> additional problems with PGPROC partitioning: we need to account
> additional padding pages in 'PGProcShmemSize' (using the same logic as
> with fplocks) and we should not call 'MemSet(ptr, 0, ...)' prior to
> partitions mapping call (otherwise it will be mapped to current node).
> Here is a potential change, which tries to address these problems:
> https://github.com/Lerm/postgres/commit/
> eaf12776f59ff150735d0f187595fc8ce3f0a872.
>
So you're saying pgproc_partition_init() should not do just this
ptr = (char *) ptr + num_procs * sizeof(PGPROC);
but align the pointer to numa_page_size too? Sounds reasonable.
Yeah, PGProcShmemSize() should have added huge pages for each partition,
just like FastPathLockShmemSize(). Seems like a bug.
I don't think the memset() is a problem. Yes, it might map it to the
current node, but so what - the numa_tonode_memory() will just move it
to the correct one.
> There are also some potential problems with buffers distribution between
> nodes. I have a feeling that current logic in
> 'buffer_partitions_prepare' does not work correctly if number of buffers
> is enough to cover just a single partition per node, but total number of
> nodes is below MIN_BUFFER_PARTITIONS (i.e. 2 or 3). In this case we will
> set 'numa_can_partition' to 'true', but will allocate 2 partitions per
> node (so, 4 or 6 partitions in total), while we can fill just 2 or 3
> partition and leaving remaining partitions empty. This should violate
> the last assert check, as last partition will get zero buffers in this
> case. Another issue is related to the usage of 1GB pages, as minimal
> size for buffers partitions is limited by the minimal number of buffer
> descriptors in a single page. For 2MB pages this gives 2097152 / 64 * 8K
> = 256M as minimal size for partition, but for 1GB page the minimal size
> is equal to 1GB / 64 * 8K = 128GB. So, if we assume 4 as minimal number
> of partitions, then for 2MB pages we need just 1GB for shared_buffers to
> enable partitioning (which seems a perfectly fine minimal limit for most
> cases), but for 1GB pages we need to allocate at least 512GB to allow
> buffers partitioning. Certainly, 1GB pages are usually used on large
> machines with large number of buffers allocated, but still it may be
> useful to allow configurations with 32GB or 64GB buffer cache to use
> both 1GB pages and buffers partitioning at the same time. However, I
> don't see an easy way to achieve this with the current logic. We either
> need to allow usage of different page sizes here (i.e. 2MB for
> descriptors and 1GB for buffers) or combine both buffers and its
> descriptors in a single object (i.e. 'buffer chunk', which cover enough
> buffers and their descriptors to fit into one or several memory pages),
> effectively replacing both buffers and descriptors arrays with an array
> of such 'chunks'. The latter solution may also help with dynamic buffer
> cache resizing (as we may just add additional 'chunks' in this case) and
> also increase TLB-hits with 1GB page (as both descriptor and its buffer
> will be likely located in the same page). However, both these changes
> seems to be quite large.
>
I'll look at handling the case with shared_buffers being too small to
allow partitioning. There well might be a bug. We should simply disable
partitioning in such cases.
As for 1GB huge pages, I don't see a good way to support configurations
with small buffers in these cases. To me it seems acceptable to say that
if you want 1GB huge pages, you should have a lot of memory and shared
buffers large enough.
I'm not against supporting such systems, if we can come up with a good
partitioning scheme. When I tried to come up with a scheme like that, it
always came with a substantial complexity & cost. The main challenge was
that it forced splitting the array of buffer descriptors, similarly to
what the PGPROC partitioning does. And that made buffer access so much
more complex / expensive it seemed not worth it. I was worried about
impact on systems without NUMA partitioning.
> I've tried also to run some benchmarks on my server: I've got some
> improvements in 'pgbench/tpcb-like'results - about 8%, but only with
> backends pinning to NUMA node (i.e. adjusting your previous pinning
> patch to 'debug_numa' GUC: https://github.com/Lerm/postgres/
> commit/5942a3e12c7c501aa9febb63972a039e7ce00c20). For 'select-only'
> scenario the gain is more substantial (about 15%), but these tests are
> tricky, as they are more sensitive to other server settings and specific
> functions layout in compiled code, so they need more checks.
>
What kind of hardware was that? What/how many cpus, NUMA nodes, how much
memory, what storage?
FWIW the main purpose of these patches was not so much throughput
improvement, but making the behavior more stable / consistent.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-13 18:34 Alexey Makhmutov <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 89+ messages in thread
From: Alexey Makhmutov @ 2025-10-13 18:34 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
On 10/13/25 14:09, Tomas Vondra wrote:
> I'm not sure I understand. Are you suggesting there's a bug in the
patch, the kernel, or somewhere else?
We need to ensure that both addr and (addr + size) are aligned to the
page size of the target mapping during 'numa_tonode_memory' invocation,
otherwise it may produce unexpected results.
> But this is exactly why (with hugepages) the code aligns everything
to huge page boundary, and sizes everything as a multiple of huge page.
At least I think so. Maybe I remember wrong?
I assume that there are places in the current patch, which could perform
such unaligned mapping. See below for samples.
> Can you actually demonstrate this?
This issue is related to the calculation of partition size for buffer
descriptors in case we have multiple partitions per node. Currently we
ensure that each node gets number of buffers, which fits into whole
memory pages, but if we have several partitions per node, then there is
no guarantee that partition size will be properly aligned for
descriptors. We could observe this problem only if we have multiple
partitions per node and with MIN_BUFFER_PARTITIONS equal to 4, this
issue can potentially affect only configurations with 2 or 3 nodes.
Two examples here: first, let's assume we want to have shared_buffers
set to 32GB with 3 NUMA nodes and 2MB pages. The NBuffers will be
4,194,304, min_node_buffers will be 32,768 and num_partitions_per_node
will be 2 (so, 6 partitions in total). NBuffers/min_node_buffers = 128,
so the nearest multiplier for min_node_buffers which allow us to cover
all buffers with 3 nodes is 43 (42*3 = 126, 43*3 = 129). The
num_buffers_per_node is 43*min_node_buffers and it is aligned to page
size, but we need to split it between two partitions, so each gets
41.5*min_node_buffers buffers. This still allow us to split buffers
itself by page boundary, but descriptor partitions will be split just in
the middle of the page. Here is the log for such configuration:
NUMA: buffers 4194304 partitions 6 num_nodes 3 per_node 2
buffers_per_node 1409024 (min 32768)
NUMA: buffer 0 node 0 partition 0 buffers 704512 first 0 last 704511
NUMA: buffer 1 node 0 partition 1 buffers 704512 first 704512 last 1409023
NUMA: buffer 2 node 1 partition 0 buffers 704512 first 1409024 last 2113535
NUMA: buffer 3 node 1 partition 1 buffers 704512 first 2113536 last 2818047
NUMA: buffer 4 node 2 partition 0 buffers 688128 first 2818048 last 3506175
NUMA: buffer 5 node 2 partition 1 buffers 688128 first 3506176 last 4194303
NUMA: buffer_partitions_init: 0 => 0 buffers 704512 start 0x7ff7c8c00000
end 0x7ff920c00000 (size 5771362304)
NUMA: buffer_partitions_init: 0 => 0 descriptors 704512 start
0x7ff7b8a00000 end 0x7ff7bb500000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 1 => 0 buffers 704512 start 0x7ff920c00000
end 0x7ffa78c00000 (size 5771362304)
NUMA: buffer_partitions_init: 1 => 0 descriptors 704512 start
0x7ff7bb500000 end 0x7ff7be000000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 2 => 1 buffers 704512 start 0x7ffa78c00000
end 0x7ffbd0c00000 (size 5771362304)
NUMA: buffer_partitions_init: 2 => 1 descriptors 704512 start
0x7ff7be000000 end 0x7ff7c0b00000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 3 => 1 buffers 704512 start 0x7ffbd0c00000
end 0x7ffd28c00000 (size 5771362304)
NUMA: buffer_partitions_init: 3 => 1 descriptors 704512 start
0x7ff7c0b00000 end 0x7ff7c3600000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 4 => 2 buffers 688128 start 0x7ffd28c00000
end 0x7ffe78c00000 (size 5637144576)
NUMA: buffer_partitions_init: 4 => 2 descriptors 688128 start
0x7ff7c3600000 end 0x7ff7c6000000 (size 44040192)
NUMA: buffer_partitions_init: 5 => 2 buffers 688128 start 0x7ffe78c00000
end 0x7fffc8c00000 (size 5637144576)
NUMA: buffer_partitions_init: 5 => 2 descriptors 688128 start
0x7ff7c6000000 end 0x7ff7c8a00000 (size 44040192)
Another example: 2 nodes and 15872MB shared_buffers. Again,
NBuffers/min_node_buffers=62, so num_buffers_per_node is
31*min_node_buffers, which gives each partition 15.5*min_node_buffers.
Here is the log output:
NUMA: buffers 2031616 partitions 4 num_nodes 2 per_node 2
buffers_per_node 1015808 (min 32768)
NUMA: buffer 0 node 0 partition 0 buffers 507904 first 0 last 507903
NUMA: buffer 1 node 0 partition 1 buffers 507904 first 507904 last 1015807
NUMA: buffer 2 node 1 partition 0 buffers 507904 first 1015808 last 1523711
NUMA: buffer 3 node 1 partition 1 buffers 507904 first 1523712 last 2031615
NUMA: buffer_partitions_init: 0 => 0 buffers 507904 start 0x7ffbf9c00000
end 0x7ffcf1c00000 (size 4160749568)
NUMA: buffer_partitions_init: 0 => 0 descriptors 507904 start
0x7ffbf1e00000 end 0x7ffbf3d00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 1 => 0 buffers 507904 start 0x7ffcf1c00000
end 0x7ffde9c00000 (size 4160749568)
NUMA: buffer_partitions_init: 1 => 0 descriptors 507904 start
0x7ffbf3d00000 end 0x7ffbf5c00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 2 => 1 buffers 507904 start 0x7ffde9c00000
end 0x7ffee1c00000 (size 4160749568)
NUMA: buffer_partitions_init: 2 => 1 descriptors 507904 start
0x7ffbf5c00000 end 0x7ffbf7b00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 3 => 1 buffers 507904 start 0x7ffee1c00000
end 0x7fffd9c00000 (size 4160749568)
NUMA: buffer_partitions_init: 3 => 1 descriptors 507904 start
0x7ffbf7b00000 end 0x7ffbf9a00000 (size 32505856)
mbind: Invalid argument
> So you're saying pgproc_partition_init() should not do just this
> ptr = (char *) ptr + num_procs * sizeof(PGPROC);
> but align the pointer to numa_page_size too? Sounds reasonable.
Yes, that's exactly my point, otherwise we could violate the alignment
rule for 'numa_tonode_memory'. Here is an extraction from the log for
system with 2 nodes, 2000 max_connections and 2MB pages:
NUMA: pgproc backends 2056 num_nodes 2 per_node 1028
NUMA: pgproc_init_partition procs 0x7fffe7800000 endptr 0x7fffe78d2d20
num_procs 1028 node 0
mbind: Invalid argument
NUMA: pgproc_init_partition procs 0x7fffe7a00000 endptr 0x7fffe7ad2d20
num_procs 1028 node 1
mbind: Invalid argument
NUMA: pgproc_init_partition procs 0x7fffe7c00000 endptr 0x7fffe7c07cb0
num_procs 38 node -1
mbind: Invalid argument
mbind: Invalid argument
> I don't think the memset() is a problem. Yes, it might map it to the
current node, but so what - the numa_tonode_memory() will just move it
to the correct one.
Well, the 'numa_tonode_memory' call does not move pages to the target
node. It just sets the policy for mapping, so system will actually try
to provide page from the correct node once we touch it. However, if the
page is already faulted, then it won't be affected by this mapping, so
that's why it works faster compared to 'numa_move_pages'. As stated in
libnuma documentation:
* numa_tonode_memory() put memory on a specific node. The constraints
described for numa_interleave_memory() apply here too.
* numa_interleave_memory() interleaves size bytes of memory page by
page from start on nodes specified in nodemask. <...> This is a lower
level function to interleave allocated but not yet faulted in memory.
Not yet faulted in means the memory is allocated using mmap(2) or
shmat(2), but has not been accessed by the current process yet. <...>
If the numa_set_strict() flag is true then the operation will cause a
numa_error if there were already pages in the mapping that do not follow
the policy.
I assume, that for the regular page kernel may rebalance memory in the
future (not immediately), but not for hugepages. So, we really don't
want to touch the memory area before we call the 'numa_tonode_memory'.
This can be easily tested with the simple program:
#include <stdio.h>
#include <numa.h>
#include <sys/mman.h>
#include <linux/mman.h>
#define MAP_SIZE 2*1024*1024
int main(int argc, char** argv) {
void* ptr1 = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED
| MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, -1, 0);
void* ptr2 = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED
| MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, -1, 0);
/* Fault first page */
memset(ptr1, 1, MAP_SIZE);
/* Move to node 1 */
numa_tonode_memory(ptr1, MAP_SIZE, 1);
numa_tonode_memory(ptr2, MAP_SIZE, 1);
/* Fault second page */
memset(ptr2, 1, MAP_SIZE);
/* Wait */
printf("ptr1=%lx\nptr2=\%lx\nPress Enter to continue...\n",ptr1,ptr2);
getchar();
munmap(ptr2, MAP_SIZE);
munmap(ptr1, MAP_SIZE);
return 0;
}
Running it on the first node:
# gcc -o test_mem test_mem.c -lnuma
# taskset -c 0 ./test_mem
ptr1=7ffff7a00000
ptr2=7ffff7800000
Press Enter to continue...
From another terminal:
# grep huge /proc/`pgrep test_mem`/numa_maps
7ffff7800000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1 N1=1
kernelpagesize_kB=2048
7ffff7a00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1 N0=1
kernelpagesize_kB=2048
So, while policy (bind:1) is set for both mappings, but only the second
one (which was not touched before the 'numa_tonode_memory' invocation)
is actualy located on node 1 rather than 0.
> What kind of hardware was that? What/how many cpus, NUMA nodes, how
much memory, what storage?
Of course, that's valid question. I probably should not have commented
on performance side without providing full data, while I was still
trying to measure it and it was just preliminary runs. Sorry for that.
Thanks,
Alexey
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-15 17:02 Tomas Vondra <[email protected]>
parent: Alexey Makhmutov <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-10-15 17:02 UTC (permalink / raw)
To: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
Hi,
Here's an updated patch series, addressing (some) of the issues. I've
kept the changes in separate patches, to make the changes easier to
review and discuss.
On 10/13/25 20:34, Alexey Makhmutov wrote:
> On 10/13/25 14:09, Tomas Vondra wrote:
>
>> I'm not sure I understand. Are you suggesting there's a bug in the
> patch, the kernel, or somewhere else?
>
> We need to ensure that both addr and (addr + size) are aligned to the
> page size of the target mapping during 'numa_tonode_memory' invocation,
> otherwise it may produce unexpected results.
>
Hmm. The libnuma docs about numa_intereave_memory says:
... The size argument will be rounded up to a multiple of the system
page size. ...
Which I interpreted that it does all the necessary rounding. But if this
ignores huge pages (i.e. "system page size" is 4K, not a HP size), then
aligning the size explicitly is would be needed.
This would be pretty annoying, though. It'd mean we can't rely on any
rounding done by libnuma, at least for code that might use huge pages.
Nevertheless, the updated patches should address both cases ...
>> But this is exactly why (with hugepages) the code aligns everything to
> huge page boundary, and sizes everything as a multiple of huge page. At
> least I think so. Maybe I remember wrong?
>
> I assume that there are places in the current patch, which could perform
> such unaligned mapping. See below for samples.
>
>> Can you actually demonstrate this?
>
> This issue is related to the calculation of partition size for buffer
> descriptors in case we have multiple partitions per node. Currently we
> ensure that each node gets number of buffers, which fits into whole
> memory pages, but if we have several partitions per node, then there is
> no guarantee that partition size will be properly aligned for
> descriptors. We could observe this problem only if we have multiple
> partitions per node and with MIN_BUFFER_PARTITIONS equal to 4, this
> issue can potentially affect only configurations with 2 or 3 nodes.
>
> Two examples here: first, let's assume we want to have shared_buffers
> set to 32GB with 3 NUMA nodes and 2MB pages. The NBuffers will be
> 4,194,304, min_node_buffers will be 32,768 and num_partitions_per_node
> will be 2 (so, 6 partitions in total). NBuffers/min_node_buffers = 128,
> so the nearest multiplier for min_node_buffers which allow us to cover
> all buffers with 3 nodes is 43 (42*3 = 126, 43*3 = 129). The
> num_buffers_per_node is 43*min_node_buffers and it is aligned to page
> size, but we need to split it between two partitions, so each gets
> 41.5*min_node_buffers buffers. This still allow us to split buffers
> itself by page boundary, but descriptor partitions will be split just in
> the middle of the page. Here is the log for such configuration:
> NUMA: buffers 4194304 partitions 6 num_nodes 3 per_node 2
> buffers_per_node 1409024 (min 32768)
> ...
I see. I was really puzzled how could a node get chunk of buffers that's
not a multiple of page size, because min_node_buffers was meant to
guarantee that. But now I realize it's not about per-node buffers, it's
about individual partitions.
Initially I thought the right way to fix this is to use min_node_buffers
for each partitions, not for nodes. But that would increase the amount
of memory needed for NUMA partitioning to work. I practice that wouldn't
be an issue, because it'd still be only ~1GB (with 2MB huge pages), and
the relevant systems will have way more.
But then I realized it's we don't need to map the partitions one by one.
We can simply map all partitions for the whole NUMA node at once, and
then we don't have this problem at all.
The attached 0007 patch does this to fix the issue. And I just noticed
this is pretty much exactly how you fixed this in your commit ee8b360.
The last partition may still not have the size aligned, though, because
may not be a multiple of min_node_buffers.
>
> Another example: 2 nodes and 15872MB shared_buffers. Again, NBuffers/
> min_node_buffers=62, so num_buffers_per_node is 31*min_node_buffers,
> which gives each partition 15.5*min_node_buffers. Here is the log output:
> NUMA: buffers 2031616 partitions 4 num_nodes 2 per_node 2
> buffers_per_node 1015808 (min 32768)
> ...
> mbind: Invalid argument
> NUMA: buffer_partitions_init: 3 => 1 buffers 507904 start 0x7ffee1c00000
> end 0x7fffd9c00000 (size 4160749568)
> NUMA: buffer_partitions_init: 3 => 1 descriptors 507904 start
> 0x7ffbf7b00000 end 0x7ffbf9a00000 (size 32505856)
> mbind: Invalid argument
>
>> So you're saying pgproc_partition_init() should not do just this
>> ptr = (char *) ptr + num_procs * sizeof(PGPROC);
>> but align the pointer to numa_page_size too? Sounds reasonable.
>
> Yes, that's exactly my point, otherwise we could violate the alignment
> rule for 'numa_tonode_memory'. Here is an extraction from the log for
> system with 2 nodes, 2000 max_connections and 2MB pages:
Should be fixed by 0010 by explicitly aligning the size like this. It's
a bit more extensive than your eaf1277.
BTW what's the mbind failures about? Is that something we check, at
least in memory
>
>> I don't think the memset() is a problem. Yes, it might map it to the
> current node, but so what - the numa_tonode_memory() will just move it
> to the correct one.
>
> Well, the 'numa_tonode_memory' call does not move pages to the target
> node. It just sets the policy for mapping, so system will actually try
> to provide page from the correct node once we touch it. However, if the
> page is already faulted, then it won't be affected by this mapping, so
> that's why it works faster compared to 'numa_move_pages'. As stated in
> libnuma documentation:
> * numa_tonode_memory() put memory on a specific node. The constraints
> described for numa_interleave_memory() apply here too.
> * numa_interleave_memory() interleaves size bytes of memory page by
> page from start on nodes specified in nodemask. <...> This is a lower
> level function to interleave allocated but not yet faulted in memory.
> Not yet faulted in means the memory is allocated using mmap(2) or
> shmat(2), but has not been accessed by the current process yet. <...>
> If the numa_set_strict() flag is true then the operation will cause a
> numa_error if there were already pages in the mapping that do not follow
> the policy.
>
Point taken. The 0009 fixes this by moving the MemSet() to after the
partitioning. At that point the policy is already set.
There's a couple more fixes. 0008 improves handling of cases that don't
allow NUMA partitioning (like when shared_buffers are too small). 0011
adds the missing padding to PGProcShmemSize, which you also fixed in one
of your commits.
0012 reduces logging in clock-sweep balancing, which on idle systems was
annoyingly verbose.
I keps 0006 separate for now. It got broken by 5e89985928, and the
conflicts were fairly extensive. Better keep it separate a bit longer.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20251015-0001-NUMA-shared-buffers-partitioning.patch (41.7K, ../../[email protected]/2-v20251015-0001-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 81749da92eb8e66fce0f3a94059b9a0dafa8bca6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251015 01/12] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
This allows partitioning clock-sweep in a later patch, with one clock
hand per partition.
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 26 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 627 +++++++++++++++++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 23 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 859 insertions(+), 16 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..fb9003c011e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,26 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 3df04c98959..8a0a4bd5cd6 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -777,3 +779,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..f51c7db7855 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +35,23 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
/*
* Data Structures:
@@ -70,19 +98,87 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
- /* Align descriptors to a cacheline boundary. */
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
+
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +208,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -148,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -175,5 +284,505 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %zd)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
+}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index b176d5130e4..82acead2e47 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -906,6 +906,16 @@
boot_val => 'true',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'sync_replication_slots', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Enables a physical standby to synchronize logical failover replication slots from the primary server.',
variable => 'sync_replication_slots',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 00c8376cf4d..d17ee9ca861 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index c1206a46aba..80b9d8d012d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -275,10 +275,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -288,7 +288,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -321,6 +321,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -484,4 +485,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3f37b294af6..43ec9f4cf0f 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -146,6 +146,28 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int numa_node; /* NUMA node (-1 no node) */
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
@@ -320,6 +342,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 3368a43a338..8ee0e7d211c 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -106,6 +109,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -128,4 +161,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5290b91e83e..b02d117c374 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.0
[text/x-patch] v20251015-0002-NUMA-clockweep-partitioning.patch (35.6K, ../../[email protected]/3-v20251015-0002-NUMA-clockweep-partitioning.patch)
download | inline diff:
From dc30d0de0c048e3ca572614c2278ce57158a4a1a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v20251015 02/12] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 11 +
src/backend/storage/buffer/bufmgr.c | 186 +++++----
src/backend/storage/buffer/freelist.c | 353 ++++++++++++++++--
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 503 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index fb9003c011e..6676e807034 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -16,7 +16,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8a0a4bd5cd6..c9dfc8a1b82 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -839,12 +847,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -860,6 +878,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index f51c7db7855..dd9f51529b4 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -716,6 +716,17 @@ BufferPartitionGet(int idx, int *node, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ if (num_partitions)
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
/* XXX the GUC hooks should probably be somewhere else? */
bool
check_debug_numa(char **newval, void **extra, GucSource source)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index edf17ce3ea1..aa209eddaab 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3614,33 +3614,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3668,25 +3664,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3698,17 +3685,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3716,11 +3703,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3740,9 +3727,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3756,15 +3743,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3772,9 +3760,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3784,7 +3772,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3792,10 +3780,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3822,7 +3810,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3847,20 +3835,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3874,7 +3862,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3904,8 +3892,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7fe34d3ef4c..952440fd9e5 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,34 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +132,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +144,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +152,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +185,117 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
+ */
+static int
+calculate_partition_index(void)
+{
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +367,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +475,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +522,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +596,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +606,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +625,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +654,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +667,44 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
@@ -802,3 +1067,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 80b9d8d012d..67f2afc623d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -443,7 +443,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -489,5 +491,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 43ec9f4cf0f..67f07d10e48 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -356,6 +356,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b02d117c374..33a9721cef3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.0
[text/x-patch] v20251015-0003-NUMA-clocksweep-allocation-balancing.patch (25.3K, ../../[email protected]/4-v20251015-0003-NUMA-clocksweep-allocation-balancing.patch)
download | inline diff:
From 38072e13f35662a953f9439c8093a7c72937564d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 10 Sep 2025 18:56:28 +0200
Subject: [PATCH v20251015 03/12] NUMA: clocksweep allocation balancing
If backends only allocate buffers from the "local" partition, this could
cause significant misbalance - some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 6676e807034..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -22,7 +22,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index c9dfc8a1b82..f6831f60b9e 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -795,6 +797,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -826,6 +830,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -847,11 +857,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -860,8 +876,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -890,6 +914,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index aa209eddaab..13258200526 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3918,6 +3918,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 952440fd9e5..73e57e13ba1 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -132,7 +168,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -144,7 +206,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -291,11 +353,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -367,7 +477,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -475,6 +585,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -501,6 +829,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -695,7 +1024,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1070,8 +1413,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1079,11 +1424,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 67f2afc623d..4ae0c32dfe5 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -443,6 +443,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 67f07d10e48..1971f0d9731 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -357,11 +357,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.0
[text/x-patch] v20251015-0004-NUMA-weighted-clocksweep-balancing.patch (5.1K, ../../[email protected]/5-v20251015-0004-NUMA-weighted-clocksweep-balancing.patch)
download | inline diff:
From 366202b656fb63400e0a1ccde29137ddab846198 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251015 04/12] NUMA: weighted clocksweep balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 73e57e13ba1..2b4c853745d 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -642,6 +642,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -668,16 +682,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -685,8 +710,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -749,6 +780,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -756,7 +791,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -770,22 +805,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.0
[text/x-patch] v20251015-0005-NUMA-partition-PGPROC.patch (48.7K, ../../[email protected]/6-v20251015-0005-NUMA-partition-PGPROC.patch)
download | inline diff:
From 8ac90c354e3305e7ce714077bf8c5ac179a5013c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 8 Sep 2025 13:11:02 +0200
Subject: [PATCH v20251015 05/12] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 ++++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 +--
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 532 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 705 insertions(+), 70 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index f6831f60b9e..a859962f5f8 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -932,3 +935,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 33369fbe23a..afa74466006 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..011fecfc58b 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index dd9f51529b4..2fd7f937ffb 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -766,6 +766,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 2b4c853745d..314ccbe4f93 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -469,7 +469,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 4cc7f645c31..d01f486876d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 96f29aafc39..70ccfebef55 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,32 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -76,8 +87,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -90,6 +101,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +134,36 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
return size;
}
@@ -129,6 +188,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -140,12 +253,16 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
return size;
}
@@ -191,7 +308,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -210,6 +327,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -224,6 +344,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,19 +370,104 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +500,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,9 +648,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -435,7 +714,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -646,7 +969,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1049,7 +1372,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1098,7 +1421,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1988,7 +2311,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2386,168 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 33a9721cef3..8b0d091c6c0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1878,6 +1878,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.0
[text/x-patch] v20251015-0006-fix-StrategyGetBuffer.patch (6.6K, ../../[email protected]/7-v20251015-0006-fix-StrategyGetBuffer.patch)
download | inline diff:
From 2fd75d9fcd58cab66cf75b1b39eb3b477ba9b311 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251015 06/12] fix: StrategyGetBuffer
StrategyGetBuffer is expected to scan all clock-sweep partitions, not
just the "local" one. So start at the optimal one (as calculated by
ChooseClockSweep), and then advance to the next one in a round-robin
way, until we find a free / unpinned buffer.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 314ccbe4f93..7f241301dcb 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -169,6 +169,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -203,10 +206,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -428,7 +430,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -482,37 +485,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -540,7 +575,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -558,7 +593,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.0
[text/x-patch] v20251015-0007-fix-map-all-buffer-partitions-for-NUMA-nod.patch (4.3K, ../../[email protected]/8-v20251015-0007-fix-map-all-buffer-partitions-for-NUMA-nod.patch)
download | inline diff:
From d6bc867374a1195d2426de629b0ffd1f22b040a9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 15:03:48 +0200
Subject: [PATCH v20251015 07/12] fix: map all buffer partitions for NUMA node
at once
Don't map individual partitions, but all partitions for the whole NUMA
node at once. This means we don't need to worry about memory pages that
span two partitions (min_node_buffers applies to the whole node). That
might be causing problems with huge pages, as reported by Alexey
Makhmutov.
Discussion: [email protected]
---
src/backend/storage/buffer/buf_init.c | 42 ++++++++++++++++++---------
1 file changed, 29 insertions(+), 13 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 2fd7f937ffb..65f45346bd1 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -460,7 +460,7 @@ buffer_partitions_prepare(void)
numa_can_partition = true; /* assume we can allocate to nodes */
if (numa_nodes > max_nodes)
{
- elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ elog(LOG, "shared buffers too small for %d nodes (max nodes %d)",
numa_nodes, max_nodes);
numa_can_partition = false;
}
@@ -630,8 +630,8 @@ buffer_partitions_init(void)
* memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
* NUMA nodes. And we don't want that.
*
- * But even with huge pages it seems like a good idea to not have mapping
- * for each page.
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
*
* So we always assign a larger contiguous chunk of buffers to the same
* NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
@@ -649,35 +649,51 @@ buffer_partitions_init(void)
* We need to account for partitions being of different length, when the
* NBuffers is not nicely divisible. To do that we keep track of the start
* of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
*/
buffers_ptr = BufferBlocks;
descriptors_ptr = (char *) BufferDescriptors;
- for (int i = 0; i < numa_partitions; i++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[i];
char *startptr,
*endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
/* first map buffers */
startptr = buffers_ptr;
- endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
buffers_ptr = endptr; /* start of the next partition */
- elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %zd)",
- i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
- pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ pg_numa_move_to_node(startptr, endptr, n);
/* now do the same for buffer descriptors */
startptr = descriptors_ptr;
- endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
descriptors_ptr = endptr;
- elog(DEBUG1, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %zd)",
- i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
- pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ pg_numa_move_to_node(startptr, endptr, n);
}
/* we should have consumed the arrays exactly */
--
2.51.0
[text/x-patch] v20251015-0008-fix-handle-disabled-NUMA-partitioning-of-b.patch (2.9K, ../../[email protected]/9-v20251015-0008-fix-handle-disabled-NUMA-partitioning-of-b.patch)
download | inline diff:
From 963e48dadc3ecc6941ee8e70d0f6d84d13ceb276 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 17:42:23 +0200
Subject: [PATCH v20251015 08/12] fix: handle disabled NUMA-partitioning of
buffers
We still partition the buffers, because we want to partition the
clock-sweep (and that relies on partitions). But we don't map the
partitions to NUMA nodes.
The NUMA partitioning may be disabled for a number of reasons. The build
may not have libnuma enabled, debug_numa may not include "buffers"
and/or the shared buffers are too small (especially with huge pages).
---
src/backend/storage/buffer/buf_init.c | 25 +++++++++++++++++++++----
1 file changed, 21 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 65f45346bd1..d0efa102d82 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -301,6 +301,10 @@ BufferGetNode(Buffer buffer)
if (numa_buffers_per_node == -1)
return 0;
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
return (buffer / numa_buffers_per_node);
}
@@ -460,10 +464,15 @@ buffer_partitions_prepare(void)
numa_can_partition = true; /* assume we can allocate to nodes */
if (numa_nodes > max_nodes)
{
- elog(LOG, "shared buffers too small for %d nodes (max nodes %d)",
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
numa_nodes, max_nodes);
numa_can_partition = false;
}
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
/*
* We know we can partition to the desired number of nodes, now it's time
@@ -483,8 +492,16 @@ buffer_partitions_prepare(void)
/*
* Finally, calculate how many buffers we'll assign to a single NUMA node.
- * If we have only a single node, or can't map to that many nodes, just
- * take a "fair share" of buffers.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
*
* XXX In both cases the last node can get fewer buffers.
*/
@@ -599,7 +616,7 @@ buffer_partitions_init(void)
Assert((num_buffers > 0) && (num_buffers <= part_buffers));
/* XXX we should get the actual node ID from the mask */
- if ((numa_flags & NUMA_BUFFERS) != 0)
+ if (numa_can_partition)
part->numa_node = n;
else
part->numa_node = -1;
--
2.51.0
[text/x-patch] v20251015-0009-fix-move-memset-after-PGPROC-partitioning.patch (1.4K, ../../[email protected]/10-v20251015-0009-fix-move-memset-after-PGPROC-partitioning.patch)
download | inline diff:
From 30a104a5b3e9567f2410b75d5fefb57d2144e606 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 16:33:55 +0200
Subject: [PATCH v20251015 09/12] fix: move memset after PGPROC partitioning
The memset faults the pages into memory, which interferes with the NUMA
(see e.g. the requirements for numa_interleave_memory). Reported by
Alexey Makhmutov.
Discussion: [email protected]
---
src/backend/storage/lmgr/proc.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 70ccfebef55..56812a05860 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -368,8 +368,6 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
/* allprocs (array of pointers to PGPROC entries) */
procs = (PGPROC **) ptr;
ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
@@ -458,6 +456,12 @@ InitProcGlobal(void)
Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
}
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
--
2.51.0
[text/x-patch] v20251015-0010-fix-pgproc-partitioning-align-size.patch (3.2K, ../../[email protected]/11-v20251015-0010-fix-pgproc-partitioning-align-size.patch)
download | inline diff:
From e8f69161eb459f89c8befd2da052f7a6bc0598ea Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 16:52:53 +0200
Subject: [PATCH v20251015 10/12] fix: pgproc partitioning - align size
I'm not sure this is actually required. The libnuma docs say:
The size argument will be rounded up to a multiple of the system page size.
so why should it be up to us to round it like that?
Discussion: [email protected]
---
src/backend/storage/lmgr/proc.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 56812a05860..fc5e1969c36 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -392,6 +392,9 @@ InitProcGlobal(void)
Assert(numa_procs_per_node > 0);
Assert(numa_nodes > 0);
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
/*
* Now initialize the PGPROC partition registry with one partition
* per NUMA node (and then one extra partition for auxiliary procs).
@@ -401,9 +404,6 @@ InitProcGlobal(void)
/* the last NUMA node may get fewer PGPROC entries, but meh */
node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
- /* make sure to align the PGPROC array to memory page */
- ptr = (char *) TYPEALIGN(numa_page_size, ptr);
-
/* fill in the partition info */
partitions[i].num_procs = node_procs;
partitions[i].numa_node = i;
@@ -411,6 +411,9 @@ InitProcGlobal(void)
ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
total_procs += node_procs;
/* don't underflow/overflow the allocation */
@@ -425,9 +428,6 @@ InitProcGlobal(void)
*/
node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
- /* make sure to align the PGPROC array to memory page */
- ptr = (char *) TYPEALIGN(numa_page_size, ptr);
-
/* fill in the partition info */
partitions[numa_nodes].num_procs = node_procs;
partitions[numa_nodes].numa_node = -1;
@@ -2452,7 +2452,7 @@ pgproc_partitions_prepare(void)
}
/*
- * doesn't do alignment
+ *
*/
static char *
pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
@@ -2465,15 +2465,20 @@ pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
/* pointer right after this array */
ptr = (char *) ptr + num_procs * sizeof(PGPROC);
- elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
- procs_node, ptr, num_procs, node);
-
/*
* if node specified, move to node - do this before we start touching the
* memory, to make sure it's not mapped to any node yet
*/
if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
/* add pointers to the PGPROC entries to allProcs */
for (int i = 0; i < num_procs; i++)
--
2.51.0
[text/x-patch] v20251015-0011-fix-add-padding-for-pgproc-partitions.patch (847B, ../../[email protected]/12-v20251015-0011-fix-add-padding-for-pgproc-partitions.patch)
download | inline diff:
From 21ec8c1801857206947c6b08701493a05e5c82cb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 17:32:29 +0200
Subject: [PATCH v20251015 11/12] fix: add padding for pgproc partitions
same as for fast-path locks
---
src/backend/storage/lmgr/proc.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index fc5e1969c36..cd938adbc33 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -163,6 +163,11 @@ PGProcShmemSize(void)
*
* XXX It might be more painful with very large huge pages (e.g. 1GB).
*/
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
return size;
}
--
2.51.0
[text/x-patch] v20251015-0012-fix-clock-sweep-log-level.patch (1.1K, ../../[email protected]/13-v20251015-0012-fix-clock-sweep-log-level.patch)
download | inline diff:
From 1b0238f4778789cb1f8fed5be1aa0fe0099edce3 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 17:59:26 +0200
Subject: [PATCH v20251015 12/12] fix: clock-sweep log level
---
src/backend/storage/buffer/freelist.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7f241301dcb..2de5cd4439e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -769,7 +769,7 @@ StrategySyncBalance(void)
*/
if (avg_allocs < 100)
{
- elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ elog(INFO, "rebalance skipped: not enough allocations (allocs: %u)",
avg_allocs);
return;
}
@@ -783,7 +783,7 @@ StrategySyncBalance(void)
*/
if (delta_allocs < (avg_allocs * 0.1))
{
- elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ elog(INFO, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
delta_allocs, (uint32) (avg_allocs * 0.1));
return;
}
--
2.51.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-15 17:15 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 89+ messages in thread
From: Tomas Vondra @ 2025-10-15 17:15 UTC (permalink / raw)
To: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
On 10/13/25 13:09, Tomas Vondra wrote:
> On 10/13/25 01:58, Alexey Makhmutov wrote:
>> Hi Tomas,
>>
>> Thank you very much for working on this problem and the entire line of
>> patches prepared! I've tried to play with these patches a little and
>> here are some my observations and suggestions.
>>
>> In the current implementation we try to use all available NUMA nodes on
>> the machine, however it's often useful to limit the database only to a
>> set of specific nodes, so that other nodes can be used for other
>> processes. In my testing I was trying to use one node out of four for
>> the client program, so I'd liked to limit the database to the remaining
>> nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
>> start the cluster, so the obvious choice for me was to use the
>> 'numa_get_membind' function instead of 'numa_num_configured_nodes' to
>> get the list of usable nodes. However, it is much easier to work with
>> logical nodes in the [0; n] range inside the PG code, so I've decided to
>> add mapping between 'logical nodes' (0-n in PG) to a set of physical
>> nodes actually returned by 'numa_get_membind'. We may need to map number
>> in both directions, so two translation tables are allocated and filled
>> at the first usage of 'pg_numa' functions. It also seems to be a good
>> idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
>> 'numa_...' calls in it and this also allows us to hide this mapping in
>> static functions. Here is the patch, which I've used to test this idea:
>> https://github.com/Lerm/postgres/
>> commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. This idea probably
>> could be extended by adding some view to expose this mapping to the user
>> (at least for testing purposes) and allow to explicitly override this
>> mapping with a GUC setting. With such GUC setting we would be able to
>> control PG memory usage on NUMA nodes without the need for systemd
>> resource control or numactl parameters.
>>
>
> I've argued to keep this out of scope for v1, to keep it smaller and
> simpler. I'm not against adding that feature, though. If someone writes
> a patch to support this. I suppose the commit you linked is a step in
> that direction.
>
On second thought, I probably spoke too soon ...
What I wanted to keep out of scope for v1 is ability to pick NUMA nodes
from Postgres, e.g. setting a GUC to limit which NUMA nodes to use, etc.
But that's not what you proposed here, clearly. You're saying we should
find which NUMA nodes the process is allowed to run, and use those.
Instead of just using all *configured* nodes. And I agree with that.
I'll take a look at your commit 9ec625c. I'm not sure it's a good idea
to have our internal "logical" node ID, and a mapping to external node
ID values (exposed by the libnuma). I was thinking maybe we should use
just the external IDs, but it's true that'd be tricky when iterating
through nodes, etc. So maybe having such mapping is a good approach.
Another thing I wasn't sure about is checking for memory-only nodes. For
example rpi5 has a NUMA node for each 1GB of memory, and each CPU is
mapped to all those nodes. For buffers this probably does not matter,
but we probably should not use those NUMA nodes for PGPROC partitioning.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-10-31 11:57 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-10-31 11:57 UTC (permalink / raw)
To: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Andres Freund <[email protected]>
Hi,
here's a significantly reworked version of this patch series.
I had a couple discussions about these patches at pgconf.eu last week,
and one interesting suggestion was that maybe it'd be easier to the
clock-sweep partitioning first, in a NUMA-oblivious way. And then add
the NUMA stuff later.
The logic is that this way we could ignore some of the hard stuff (e.g.
handling huge page reservation), while still reducing clocksweep
contention. Which we speculated might be the main benefit anyway.
The attached patches do this.
0001 - Introduces a simplified version of the "buffer partition
registry" (think array in shmem, storing info about ranges of shared
buffer). The partitions are calculated as simple fraction of shared
buffers. There's no need to align the partitions to memory pages etc.
0002-0005 - Does the clock-sweep partitioning. I chose to keep this
split into smaller increments, to keep the patches easier to review.
0006 - Make the partitioning NUMA-aware. This used to be part of 0001,
but now it's moved on top of the clock-sweep stuff. It ensures the
partitions are properly aligned to memory pages, and all that.
0007 - PGPROC partitioning.
This made the 0001 patch much simpler/smaller - it used to be ~50kB, now
it's 15kB (and most of the complexity is in 0006).
The question however is how this performs, or how much of the benefit
was due to NUMA-awareness and how much was due to just partitioning
clock-sweep. I repeated the benchmark from [1], doing concurrent
sequential scans to put significant pressure on buffer replacements, and
I got this:
hp clients | master | sweep sweep-16 | numa numa-16
=============|==========|===================|===============
off 16 | 24 | 46 46 | 33 40
32 | 33 | 53 51 | 45 51
48 | 38 | 51 61 | 46 56
64 | 41 | 56 75 | 47 65
80 | 47 | 53 77 | 48 71
96 | 45 | 54 80 | 47 66
112 | 45 | 52 83 | 44 65
128 | 43 | 55 81 | 39 48
-------------|----------|-------------------|---------------
on 16 | 26 | 47 47 | 35 42
32 | 33 | 49 52 | 40 49
48 | 39 | 52 63 | 43 57
64 | 42 | 53 72 | 43 66
80 | 43 | 54 81 | 46 71
96 | 48 | 58 80 | 49 73
112 | 51 | 58 78 | 51 76
128 | 55 | 60 83 | 52 76
"hp" means huge pages, the compared branches are:
- master - current master
- sweep - patches up to 0005, default number of partitions (4)
- sweep-16 - patches up to 0005, 16 partitions
- numa - patches up to 0006, default number of partitions (4)
- numa-16 - patches up to 0006, 16 partitions
Compared to master, the results look like this:
hp clients | sweep sweep-16 | numa numa-16
==============|====================|================
off 16 | 192% 192% | 138% 167%
32 | 161% 155% | 136% 155%
48 | 132% 160% | 121% 145%
64 | 137% 183% | 115% 159%
80 | 113% 164% | 102% 151%
96 | 120% 177% | 104% 146%
112 | 116% 184% | 98% 144%
128 | 128% 186% | 90% 110%
--------------|--------------------|----------------
on 16 | 181% 181% | 135% 162%
32 | 148% 158% | 121% 148%
48 | 133% 161% | 110% 144%
64 | 126% 171% | 102% 157%
80 | 126% 188% | 107% 165%
96 | 121% 167% | 102% 152%
112 | 114% 153% | 100% 149%
128 | 109% 151% | 95% 138%
The attached PDF has more results for runs with somewhat modified
parameters, but the overall it's very similar to these numbers.
I think this confirms most of the benefit really comes from just
partitioning clock-sweep, and it's mostly independent of the NUMA stuff.
In fact, the NUMA partitioning is often slower. Some of this may be due
to inefficiencies in the patch (e.g. division in formula calculating the
partition index, etc.).
So I think this looks quite promising ...
There are a couple unsolved issues, though. While running the tests, I
ran into a bunch of weird issues. I saw two types of failures:
1) Bad address
-----------------------------------------------------------------------
2025-10-30 15:24:21.195 UTC [2038558] LOG: could not read blocks
114543..114558 in file "base/16384/16588": Bad address
2025-10-30 15:24:21.195 UTC [2038558] STATEMENT: SELECT * FROM t_41
OFFSET 1000000000
2025-10-30 15:24:21.195 UTC [2038523] LOG: could not read blocks
119981..119996 in file "base/16384/16869": Bad address
2025-10-30 15:24:21.195 UTC [2038523] CONTEXT: completing I/O on behalf
of process 2038464
2025-10-30 15:24:21.195 UTC [2038523] STATEMENT: SELECT * FROM t_96
OFFSET 1000000000
2025-10-30 15:24:21.195 UTC [2038492] LOG: could not read blocks
118226..118232 in file "base/16384/16478": Bad address
2025-10-30 15:24:21.195 UTC [2038492] STATEMENT: SELECT * FROM t_19
OFFSET 1000000000
2025-10-30 15:24:21.196 UTC [2038477] LOG: could not read blocks
120515..120517 in file "base/16384/16945": Bad address
2025-10-30 15:24:21.196 UTC [2038477] CONTEXT: completing I/O on behalf
of process 2038545
2025-10-30 15:24:21.196 UTC [2038477] STATEMENT: SELECT * FROM t_111
OFFSET 1000000000
-----------------------------------------------------------------------
2) Operation canceled
-----------------------------------------------------------------------
2025-10-31 10:57:21.742 UTC [2685933] LOG: could not read blocks
159..174 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.742 UTC [2685933] STATEMENT: SELECT * FROM t_3
OFFSET 1000000000
2025-10-31 10:57:21.742 UTC [2685933] LOG: could not read blocks
143..158 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.742 UTC [2685933] STATEMENT: SELECT * FROM t_3
OFFSET 1000000000
2025-10-31 10:57:21.781 UTC [2685933] ERROR: could not read blocks
143..158 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.781 UTC [2685933] STATEMENT: SELECT * FROM t_3
OFFSET 1000000000
-----------------------------------------------------------------------
I'm still not sure what's causing these, and it's happening rarely and
randomly, so it's hard to catch and reproduce. I'd welcome suggestions
what to look for / what might be the issue.
I did run the whole test under valgrind to make sure there's nothing
obviously broken, but that found no issues. Of course, it's much slower
under valgrind, so maybe it just didn't hit the issue.
I suspect the "bad address" might be just a different symptom of the
issues with reserving huge pages I already mentioned [2]. I assume
io_uring might try using huge pages internally, and then it fails
because postgres also reserves huge pages.
I have no idea what "operation canceled" might be about.
I'm not entirely sure if this affect all patches, or just the patches
with NUMA partitioning. Or if this happens with huge pages. I'll do more
runs to test specifically this.
But it does seem to be specific to io_uring - or at least the canceled
issue. I haven't seen it after switching to "worker".
[1]
https://www.postgresql.org/message-id/51e51832-7f47-412a-a1a6-b972101cc8cb%40vondra.me
[2]
https://www.postgresql.org/message-id/1d57d68d-b178-415a-ba11-be0c3714638e%40vondra.me
regards
--
Tomas Vondra
Attachments:
[application/pdf] clocksweep-results.pdf (60.4K, ../../[email protected]/2-clocksweep-results.pdf)
download
[text/x-patch] v20251101-0007-NUMA-partition-PGPROC.patch (49.2K, ../../[email protected]/3-v20251101-0007-NUMA-partition-PGPROC.patch)
download | inline diff:
From 5ecfc8c4f042219b4691e5f0770c72f55c4830ac Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 8 Sep 2025 13:11:02 +0200
Subject: [PATCH v20251101 7/7] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 96 ++-
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 ++-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 550 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 73 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 61207357e53..a859962f5f8 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -29,7 +30,8 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -932,3 +935,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 33369fbe23a..afa74466006 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..3288900bb6f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -292,7 +292,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..011fecfc58b 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index f8314a0e299..d0efa102d82 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -799,6 +799,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 341eaf55577..2de5cd4439e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -472,7 +472,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 4cc7f645c31..d01f486876d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 96f29aafc39..cd938adbc33 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,32 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -76,8 +87,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -90,6 +101,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +134,41 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -129,6 +193,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -140,12 +258,16 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
return size;
}
@@ -191,7 +313,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -210,6 +332,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -224,6 +349,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -239,21 +373,110 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +509,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
- /* Common initialization for all PGPROCs, regardless of type. */
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ total_procs += node_procs;
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,9 +657,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -435,7 +723,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -646,7 +978,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1049,7 +1381,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1098,7 +1430,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1988,7 +2320,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2395,173 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ *
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f7730ece976..be4a6334fb7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1878,6 +1878,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.0
[text/x-patch] v20251101-0006-NUMA-shared-buffers-partitioning.patch (43.6K, ../../[email protected]/4-v20251101-0006-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From cff7ec32ca1256227d4c2ea51c61986b10c0f28d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:41:26 +0100
Subject: [PATCH v20251101 6/7] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
---
.../pg_buffercache--1.6--1.7.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 44 +-
src/backend/storage/buffer/buf_init.c | 569 +++++++++++++++++-
src/backend/storage/buffer/freelist.c | 88 ++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 14 +-
src/include/storage/bufmgr.h | 4 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
11 files changed, 736 insertions(+), 68 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 2c4d560514d..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -13,6 +13,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer, -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index b0b9112fdba..61207357e53 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -814,19 +814,21 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
INT8OID, -1, 0);
@@ -850,7 +852,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -869,7 +872,7 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
@@ -887,36 +890,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
- values[4] = Int64GetDatum(complete_passes);
+ values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
- values[5] = Int32GetDatum(next_victim_buffer);
+ values[5] = Int64GetDatum(complete_passes);
nulls[5] = false;
- values[6] = Int64GetDatum(buffer_total_allocs);
+ values[6] = Int32GetDatum(next_victim_buffer);
nulls[6] = false;
- values[7] = Int64GetDatum(buffer_allocs);
+ values[7] = Int64GetDatum(buffer_total_allocs);
nulls[7] = false;
- values[8] = Int64GetDatum(buffer_total_req_allocs);
+ values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
- values[9] = Int64GetDatum(buffer_req_allocs);
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
nulls[9] = false;
- values[10] = PointerGetDatum(array);
+ values[10] = Int64GetDatum(buffer_req_allocs);
nulls[10] = false;
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 0362fda24aa..f8314a0e299 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,12 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -29,15 +35,24 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
-/* *
- * number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
/* Array of structs with information about buffer ranges */
BufferPartitions *BufferPartitionsArray = NULL;
+static void buffer_partitions_prepare(void);
static void buffer_partitions_init(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -85,25 +100,85 @@ BufferManagerShmemInit(void)
foundIOCV,
foundBufCkpt,
foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
/* allocate the partition registry first */
BufferPartitionsArray = (BufferPartitions *)
ShmemInitStruct("Buffer Partitions",
offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ mul_size(sizeof(BufferPartition), numa_partitions),
&foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -133,7 +208,10 @@ BufferManagerShmemInit(void)
{
int i;
- /* Initialize buffer partitions (calculate buffer ranges). */
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
buffer_partitions_init();
/*
@@ -172,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -201,11 +286,244 @@ BufferManagerShmemSize(void)
/* account for registry of NUMA partitions */
size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+ mul_size(sizeof(BufferPartition), numa_partitions)));
return size;
}
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
/*
* Sanity checks of buffers partitions - there must be no gaps, it must cover
* the whole range of buffers, etc.
@@ -267,33 +585,137 @@ buffer_partitions_init(void)
{
int remaining_buffers = NBuffers;
int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers
- = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
-
- BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
- for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[n];
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
- /* buffers this partition should get (last partition can get fewer) */
- int num_buffers = Min(remaining_buffers, part_buffers);
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
- remaining_buffers -= num_buffers;
+ remaining_buffers -= node_buffers;
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- buffer += num_buffers;
+ /* XXX we should get the actual node ID from the mask */
+ if (numa_can_partition)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
}
AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ char *startptr,
+ *endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
}
int
@@ -302,14 +724,21 @@ BufferPartitionCount(void)
return BufferPartitionsArray->npartitions;
}
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
{
BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -322,8 +751,82 @@ BufferPartitionGet(int idx, int *num_buffers,
/* return parameters before the partitions are initialized (during sizing) */
void
-BufferPartitionParams(int *num_partitions)
+BufferPartitionParams(int *num_partitions, int *num_nodes)
{
if (num_partitions)
- *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index c73a418defa..341eaf55577 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -124,7 +124,9 @@ typedef struct
//int __attribute__((aligned(64))) bgwprocno;
/* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
int num_partitions;
+ int num_partitions_per_node;
/* clocksweep partitions */
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
@@ -270,16 +272,72 @@ ClockSweepTick(ClockSweep *sweep)
* calculate_partition_index
* calculate the buffer / clock-sweep partition to use
*
- * use PID to determine the buffer partition
- *
- * XXX We could use NUMA node / core ID to pick partition, but we'd need
- * to handle cases with fewer nodes/cores than partitions somehow. Although,
- * maybe the balancing would handle that too.
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
*/
static int
calculate_partition_index(void)
{
- return (MyProcPid % StrategyControl->num_partitions);
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
}
/*
@@ -947,7 +1005,7 @@ StrategyShmemSize(void)
Size size = 0;
int num_partitions;
- BufferPartitionParams(&num_partitions);
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -974,9 +1032,17 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
int num_partitions;
+ int num_partitions_per_node;
num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -1011,7 +1077,8 @@ StrategyInitialize(bool init)
/* Initialize the clock sweep pointers (for all partitions) */
for (int i = 0; i < num_partitions; i++)
{
- int num_buffers,
+ int node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -1020,7 +1087,8 @@ StrategyInitialize(bool init)
pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
/* get info about the buffer partition */
- BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
/*
* FIXME This may not quite right, because if NBuffers is not a
@@ -1056,6 +1124,8 @@ StrategyInitialize(bool init)
/* initialize the partitioned clocksweep */
StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index d6fc8333850..1da2a43220a 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -906,6 +906,16 @@
boot_val => 'true',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'sync_replication_slots', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Enables a physical standby to synchronize logical failover replication slots from the primary server.',
variable => 'sync_replication_slots',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 00c8376cf4d..d17ee9ca861 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 776be11dec1..4ae0c32dfe5 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -275,10 +275,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -288,7 +288,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -490,8 +490,8 @@ extern void AtEOXact_LocalBuffers(bool isCommit);
extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
-extern void BufferPartitionParams(int *num_partitions);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4e7b1fcd4ab..510018db115 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,10 +156,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -169,6 +171,7 @@ typedef struct BufferPartition
typedef struct BufferPartitions
{
int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -346,6 +349,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 3368a43a338..8ee0e7d211c 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -106,6 +109,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -128,4 +161,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
--
2.51.0
[text/x-patch] v20251101-0005-clock-sweep-weighted-balancing.patch (5.2K, ../../[email protected]/5-v20251101-0005-clock-sweep-weighted-balancing.patch)
download | inline diff:
From 59aeeec92549e95c52446c59a1c1308d548d277b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251101 5/7] clock-sweep: weighted balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
Note: This may be more important with NUMA-aware partitioning, which
restricts the allowed sizes of partiions (especially with huge pages).
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 38f339b02c4..c73a418defa 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -619,6 +619,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -645,16 +659,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -662,8 +687,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -726,6 +757,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -733,7 +768,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -747,22 +782,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.0
[text/x-patch] v20251101-0004-clock-sweep-scan-all-partitions.patch (6.7K, ../../[email protected]/6-v20251101-0004-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From e4252a2559496947f6862ee0851a88dd2f59ea11 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251101 4/7] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 8897f2e9b0e..38f339b02c4 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -167,6 +167,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -201,10 +204,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -370,7 +372,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -424,37 +427,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -482,7 +517,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -500,7 +535,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.0
[text/x-patch] v20251101-0003-clock-sweep-balancing-of-allocations.patch (25.3K, ../../[email protected]/7-v20251101-0003-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From a2cb64b97d17bf271ec0aa9496601b1d644a94e2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:45:34 +0100
Subject: [PATCH v20251101 3/7] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 14e750beeff..2c4d560514d 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -21,7 +21,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 96eee21932f..b0b9112fdba 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -795,6 +797,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -824,6 +828,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -844,11 +854,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -857,8 +873,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -884,6 +908,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffer_allocs);
nulls[7] = false;
+ values[8] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[8] = false;
+
+ values[9] = Int64GetDatum(buffer_req_allocs);
+ nulls[9] = false;
+
+ values[10] = PointerGetDatum(array);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 15ca6f60d57..176a6e6358c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3913,6 +3913,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 1d2f97b4d1a..8897f2e9b0e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -130,7 +166,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -142,7 +204,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -233,11 +295,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -309,7 +419,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -417,6 +527,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(INFO, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(INFO, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -443,6 +771,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -627,7 +956,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1000,8 +1343,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1009,11 +1354,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1386856046c..776be11dec1 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -443,6 +443,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7052f9de57c..4e7b1fcd4ab 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -360,11 +360,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.0
[text/x-patch] v20251101-0002-clock-sweep-basic-partitioning.patch (33.9K, ../../[email protected]/8-v20251101-0002-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 1eb17c28164b30f759208b56eaa699236582c008 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:39:47 +0100
Subject: [PATCH v20251101 2/7] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 38 ++-
src/backend/storage/buffer/buf_init.c | 7 +
src/backend/storage/buffer/bufmgr.c | 186 ++++++++----
src/backend/storage/buffer/freelist.c | 283 +++++++++++++++---
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 432 insertions(+), 106 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index f1c20960b7e..14e750beeff 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -15,7 +15,13 @@ CREATE VIEW pg_buffercache_partitions AS
(partition integer, -- partition index
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index dfec1380736..96eee21932f 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -810,12 +810,20 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 1, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -836,12 +844,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -854,6 +872,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[3] = Int32GetDatum(last_buffer);
nulls[3] = false;
+ values[4] = Int64GetDatum(complete_passes);
+ nulls[4] = false;
+
+ values[5] = Int32GetDatum(next_victim_buffer);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffer_total_allocs);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_allocs);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 007f7806dc4..0362fda24aa 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -320,3 +320,10 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions)
+{
+ if (num_partitions)
+ *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e8544acb784..15ca6f60d57 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3609,33 +3609,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3663,25 +3659,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3693,17 +3680,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3711,11 +3698,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3735,9 +3722,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3751,15 +3738,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3767,9 +3755,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3779,7 +3767,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3787,10 +3775,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3817,7 +3805,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3842,20 +3830,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3869,7 +3857,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3899,8 +3887,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7fe34d3ef4c..1d2f97b4d1a 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +130,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +142,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +150,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +183,61 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * use PID to determine the buffer partition
+ *
+ * XXX We could use NUMA node / core ID to pick partition, but we'd need
+ * to handle cases with fewer nodes/cores than partitions somehow. Although,
+ * maybe the balancing would handle that too.
+ */
+static int
+calculate_partition_index(void)
+{
+ return (MyProcPid % StrategyControl->num_partitions);
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +309,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +417,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +464,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +538,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +548,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +567,10 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_partitions;
+
+ num_partitions = BufferPartitionCount();
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +588,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +601,40 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
}
else
Assert(!init);
@@ -802,3 +997,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 80b0c14831c..1386856046c 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -443,7 +443,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -489,5 +491,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 24860c6c2c4..7052f9de57c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -359,6 +359,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e24703d6688..f7730ece976 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.0
[text/x-patch] v20251101-0001-Infrastructure-for-partitioning-shared-buf.patch (15.0K, ../../[email protected]/9-v20251101-0001-Infrastructure-for-partitioning-shared-buf.patch)
download | inline diff:
From 03ed6e5116bd5769032823c59e5169b8219ad26b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251101 1/7] Infrastructure for partitioning shared buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep).
The registry is a small BufferPartitions array in shared memory, with
partitions sized to be a fair share of shared buffers. Later patches may
improve this to consider NUMA, and similar details.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
Note: This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 25 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 145 +++++++++++++++++-
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
9 files changed, 285 insertions(+), 3 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..f1c20960b7e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,25 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 3df04c98959..dfec1380736 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -777,3 +779,87 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..007f7806dc4 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -17,6 +17,11 @@
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +29,14 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/* *
+ * number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_init(void);
/*
* Data Structures:
@@ -70,7 +83,15 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ &foundParts);
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -112,6 +133,9 @@ BufferManagerShmemInit(void)
{
int i;
+ /* Initialize buffer partitions (calculate buffer ranges). */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -175,5 +199,124 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+
return size;
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers
+ = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[n];
+
+ /* buffers this partition should get (last partition can get fewer) */
+ int num_buffers = Min(remaining_buffers, part_buffers);
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index c1206a46aba..80b0c14831c 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -321,6 +321,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -484,4 +485,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b5f8f3c5d42..24860c6c2c4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -153,6 +153,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ac2da4c98cf..e24703d6688 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-04 12:10 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-11-04 12:10 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On Fri, Oct 31, 2025 at 12:57 PM Tomas Vondra <[email protected]> wrote:
>
> Hi,
>
> here's a significantly reworked version of this patch series.
>
> I had a couple discussions about these patches at pgconf.eu last week,[..]
I've just had a quick look at this and oh, my, I've started getting
into this partitioned clocksweep and that's ambitious! Yes, this
sequencing of patches makes it much more understandable. Anyway I've
spotted some things, attempted to fix some and have some basic
questions too (so small baby steps, all of this was on 4s/4 NUMA nodes
with HP on) -- the 000X refers to question/issue/bug in specific
patchset file:
0001: you mention 'debug_numa = buffers' in commitmsg, but there's
nothing there like that? it comes with 0006
0002: dunno, but wouldn't it make some educational/debugging sense to
add a debug function returning clocksweep partition index
(calculate_partition_index) for backend? (so that we know which
partition we are working on right now)
0003: those two "elog(INFO, "rebalance skipped:" should be at DEBUG2+
IMHO (they are way too verbose during runs)
0006a: Needs update - s/patches later in the patch series/patches
earlier in the patch series/
0006b: IMHO longer term, we should hide some complexity of those calls
via src/port numa shims (pg_numa_sched_cpu()?)
0006c: after GUC commit fce7c73fba4e5, apply complains with:
error: patch failed: src/backend/utils/misc/guc_parameters.dat:906
error: src/backend/utils/misc/guc_parameters.dat: patch does not apply
0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
I've thought it is going to be simpler to use to_hex() -- see 0009
attached.
0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
called pg_shm_pgproc?
0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
during start:
2025-11-04 10:02:27.055 CET [58464] DEBUG: NUMA:
pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
num_procs 2523 node 0
2025-11-04 10:02:27.057 CET [58464] DEBUG: NUMA:
pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
num_procs 2523 node 1
2025-11-04 10:02:27.059 CET [58464] DEBUG: NUMA:
pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
num_procs 2523 node 2
2025-11-04 10:02:27.061 CET [58464] DEBUG: NUMA:
pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
num_procs 2523 node 3
2025-11-04 10:02:27.062 CET [58464] DEBUG: NUMA:
pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
num_procs 38 node -1
mbind: Invalid argument
mbind: Invalid argument
mbind: Invalid argument
mbind: Invalid argument
0007d: so we probably need numa_warn()/numa_error() wrappers (this was
initially part of NUMA observability patches but got removed during
the course of action), I'm attaching 0008. With that you'll get
something a little more up to our standards:
2025-11-04 10:27:07.140 CET [59696] DEBUG:
fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
0x7f4f4d4b1660
2025-11-04 10:27:07.140 CET [59696] WARNING: libnuma: ERROR: mbind
0007e: elog DEBUG says it's pg_proc_init_partition but it's
pgproc_partition_init() actually ;)
0007f: The "mbind: Invalid argument"" issue itself with the below addition:
+elog(DEBUG1, "NUMA: fastpath_partition_init ptr %p endptr %p
num_procs %d node %d", ptr, endptr, num_procs, node);
showed this:
2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f39eea00000 endptr 0x7f39eeab1660
num_procs 2523 node 0
2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f39eec00000 endptr 0x7f39eecb1660
num_procs 2523 node 1
2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f39eee00000 endptr 0x7f39eeeb1660
num_procs 2523 node 2
2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
[..]
Meanwhile it's full hugepage size (e.g. 0x7f39eec00000−0x7f39eea00000 = 2MB)
$ grep --color 7f39ee[ace] /proc/61841/smaps
7f39ee800000-7f39eea00000 rw-s 87de00000 00:11 122710
/anon_hugepage (deleted)
7f39eea00000-7f39eec00000 rw-s 87e000000 00:11 122710
/anon_hugepage (deleted)
7f39eec00000-7f39eee00000 rw-s 87e200000 00:11 122710
/anon_hugepage (deleted)
7f39eee00000-7f39ef000000 rw-s 87e400000 00:11 122710
/anon_hugepage (deleted)
but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
0xB1660 = 726624 bytes, but if adjust blindly endptr in that
fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
(HP) it doesn't complain anymore and I get success:
2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f7bf7000000 endptr 0x7f7bf7200000
num_procs 2523 node 0
2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f7bf7200000 endptr 0x7f7bf7400000
num_procs 2523 node 1
2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f7bf7400000 endptr 0x7f7bf7600000
num_procs 2523 node 2
2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f7bf7600000 endptr 0x7f7bf7800000
num_procs 2523 node 3
2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
fastpath_partition_init ptr 0x7f7bf7800000 endptr 0x7f7bf7a00000
num_procs 38 node -1
2025-11-04 12:08:30.239 CET [62352] LOG: starting PostgreSQL
19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
0006d: I've got one SIGBUS during a call to select
pg_buffercache_numa_pages(); and it looks like that memory accessed is
simply not mapped? (bug)
Program received signal SIGBUS, Bus error.
pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
386 pg_numa_touch_mem_if_required(ptr);
(gdb) print ptr
$1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
(gdb) where
#0 pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
#1 0x0000561a672a0efe in ExecMakeFunctionResultSet
(fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
isDone=isDone@entry=0x561a97e8e5c0) at
../src/backend/executor/execSRF.c:624
[..]
Postmaster had still attached shm (visible via smaps), and if you
compare closely 0x7f4ed0200000 against sorted smaps:
7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
/anon_hugepage (deleted)
7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
/anon_hugepage (deleted)
7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
/anon_hugepage (deleted)
7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
/anon_hugepage (deleted)
7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
/anon_hugepage (deleted)
it's NOT there at all (there's no mmap region starting with
0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
aware of this new mmaped() regions and instead does simple loop over
all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
os_page_size)"?
0006e:
I'm seeking confirmation, but is this the issue we have discussed
on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
$ numactl --membind="0,1" --cpunodebind="0,1"
/usr/pgsql19/bin/pg_ctl -D /path start
still shows 4 NUMA nodes used. Current patches use
numa_num_configured_nodes(), but it says 'This count includes any
nodes that are currently DISABLED'. So I was wondering if I could help
by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
It's the same as You wrote earlier to Alexy?
> But that's not what you proposed here, clearly. You're saying we should
> find which NUMA nodes the process is allowed to run, and use those.
> Instead of just using all *configured* nodes. And I agree with that.
So are you already on it ?
> There are a couple unsolved issues, though. While running the tests, I
> ran into a bunch of weird issues. I saw two types of failures:
> 1) Bad address
> 2) Operation canceled
I did run (with io_uring) a short test(< 10min with -c 128) and didn't
get those. Could you please share specific tips/workload for
reproducing this?
That's all for today, I hope it helps a little.
-J.
Attachments:
[application/octet-stream] 0008.patch_txt (2.5K, ../../CAKZiRmzQ=jzWSz7NjHggyqpnMkZUaeO00t7rUovs+zvi_YY48w@mail.gmail.com/2-0008.patch_txt)
download
[application/octet-stream] 0009.patch_txt (1.0K, ../../CAKZiRmzQ=jzWSz7NjHggyqpnMkZUaeO00t7rUovs+zvi_YY48w@mail.gmail.com/3-0009.patch_txt)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-04 21:21 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-11-04 21:21 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On 11/4/25 13:10, Jakub Wartak wrote:
> On Fri, Oct 31, 2025 at 12:57 PM Tomas Vondra <[email protected]> wrote:
>>
>> Hi,
>>
>> here's a significantly reworked version of this patch series.
>>
>> I had a couple discussions about these patches at pgconf.eu last week,[..]
>
> I've just had a quick look at this and oh, my, I've started getting
> into this partitioned clocksweep and that's ambitious! Yes, this
> sequencing of patches makes it much more understandable. Anyway I've
> spotted some things, attempted to fix some and have some basic
> questions too (so small baby steps, all of this was on 4s/4 NUMA nodes
> with HP on) -- the 000X refers to question/issue/bug in specific
> patchset file:
>
> 0001: you mention 'debug_numa = buffers' in commitmsg, but there's
> nothing there like that? it comes with 0006
>
Right, I forgot to remove that reference.
> 0002: dunno, but wouldn't it make some educational/debugging sense to
> add a debug function returning clocksweep partition index
> (calculate_partition_index) for backend? (so that we know which
> partition we are working on right now)
>
Perhaps. I didn't need that, but it might be interesting during
development. I probably would not keep that in the final version.
> 0003: those two "elog(INFO, "rebalance skipped:" should be at DEBUG2+
> IMHO (they are way too verbose during runs)
>
Agreed.
> 0006a: Needs update - s/patches later in the patch series/patches
> earlier in the patch series/
>
Agreed.
> 0006b: IMHO longer term, we should hide some complexity of those calls
> via src/port numa shims (pg_numa_sched_cpu()?)
>
Yeah, there's definitely room for moving more of the code to src/port.
> 0006c: after GUC commit fce7c73fba4e5, apply complains with:
> error: patch failed: src/backend/utils/misc/guc_parameters.dat:906
> error: src/backend/utils/misc/guc_parameters.dat: patch does not apply
>
Will fix.
> 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
> bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
> I've thought it is going to be simpler to use to_hex() -- see 0009
> attached.
>
I don't know. I added simply because it might be useful for development,
but we probably don't want to expose these pointers at all.
> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> called pg_shm_pgproc?
>
Right. It does not belong to pg_buffercache at all, I just added it
there because I've been messing with that code already.
> 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
> during start:
>
> 2025-11-04 10:02:27.055 CET [58464] DEBUG: NUMA:
> pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
> num_procs 2523 node 0
> 2025-11-04 10:02:27.057 CET [58464] DEBUG: NUMA:
> pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
> num_procs 2523 node 1
> 2025-11-04 10:02:27.059 CET [58464] DEBUG: NUMA:
> pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
> num_procs 2523 node 2
> 2025-11-04 10:02:27.061 CET [58464] DEBUG: NUMA:
> pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
> num_procs 2523 node 3
> 2025-11-04 10:02:27.062 CET [58464] DEBUG: NUMA:
> pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
> num_procs 38 node -1
> mbind: Invalid argument
> mbind: Invalid argument
> mbind: Invalid argument
> mbind: Invalid argument
>
I'll take a look, but I don't recall seeing such errors.
> 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
> initially part of NUMA observability patches but got removed during
> the course of action), I'm attaching 0008. With that you'll get
> something a little more up to our standards:
> 2025-11-04 10:27:07.140 CET [59696] DEBUG:
> fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
> 0x7f4f4d4b1660
> 2025-11-04 10:27:07.140 CET [59696] WARNING: libnuma: ERROR: mbind
>
Not sure.
> 0007e: elog DEBUG says it's pg_proc_init_partition but it's
> pgproc_partition_init() actually ;)
>
> 0007f: The "mbind: Invalid argument"" issue itself with the below addition:
> +elog(DEBUG1, "NUMA: fastpath_partition_init ptr %p endptr %p
> num_procs %d node %d", ptr, endptr, num_procs, node);
> showed this:
> 2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f39eea00000 endptr 0x7f39eeab1660
> num_procs 2523 node 0
> 2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
> 2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f39eec00000 endptr 0x7f39eecb1660
> num_procs 2523 node 1
> 2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
> 2025-11-04 11:30:51.089 CET [61841] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f39eee00000 endptr 0x7f39eeeb1660
> num_procs 2523 node 2
> 2025-11-04 11:30:51.089 CET [61841] WARNING: libnuma: ERROR: mbind
> [..]
>
> Meanwhile it's full hugepage size (e.g. 0x7f39eec00000−0x7f39eea00000 = 2MB)
> $ grep --color 7f39ee[ace] /proc/61841/smaps
> 7f39ee800000-7f39eea00000 rw-s 87de00000 00:11 122710
> /anon_hugepage (deleted)
> 7f39eea00000-7f39eec00000 rw-s 87e000000 00:11 122710
> /anon_hugepage (deleted)
> 7f39eec00000-7f39eee00000 rw-s 87e200000 00:11 122710
> /anon_hugepage (deleted)
> 7f39eee00000-7f39ef000000 rw-s 87e400000 00:11 122710
> /anon_hugepage (deleted)
>
> but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
> 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
> fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
> (HP) it doesn't complain anymore and I get success:
> 2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f7bf7000000 endptr 0x7f7bf7200000
> num_procs 2523 node 0
> 2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f7bf7200000 endptr 0x7f7bf7400000
> num_procs 2523 node 1
> 2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f7bf7400000 endptr 0x7f7bf7600000
> num_procs 2523 node 2
> 2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f7bf7600000 endptr 0x7f7bf7800000
> num_procs 2523 node 3
> 2025-11-04 12:08:30.147 CET [62352] DEBUG: NUMA:
> fastpath_partition_init ptr 0x7f7bf7800000 endptr 0x7f7bf7a00000
> num_procs 38 node -1
> 2025-11-04 12:08:30.239 CET [62352] LOG: starting PostgreSQL
> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>
Hmm, so it seems like another hugepage-related issue. The mbind manpage
says this about "len":
EINVAL An invalid value was specified for flags or mode; or addr + len
was less than addr; or addr is not a multiple of the system page size.
I don't think that requires (addr+len) to be a multiple of page size,
but maybe that is required.
> 0006d: I've got one SIGBUS during a call to select
> pg_buffercache_numa_pages(); and it looks like that memory accessed is
> simply not mapped? (bug)
>
> Program received signal SIGBUS, Bus error.
> pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> 386 pg_numa_touch_mem_if_required(ptr);
> (gdb) print ptr
> $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
> (gdb) where
> #0 pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> #1 0x0000561a672a0efe in ExecMakeFunctionResultSet
> (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
> argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
> isDone=isDone@entry=0x561a97e8e5c0) at
> ../src/backend/executor/execSRF.c:624
> [..]
>
> Postmaster had still attached shm (visible via smaps), and if you
> compare closely 0x7f4ed0200000 against sorted smaps:
>
> 7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
> /anon_hugepage (deleted)
> 7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
> /anon_hugepage (deleted)
> 7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
> /anon_hugepage (deleted)
> 7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
> /anon_hugepage (deleted)
> 7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
> /anon_hugepage (deleted)
>
> it's NOT there at all (there's no mmap region starting with
> 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
> aware of this new mmaped() regions and instead does simple loop over
> all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
> os_page_size)"?
>
I'm confused. How could that mapping be missing? Was this with huge
pages / how many did you reserve on the nodes? Maybe there were not
enough huge pages left on one of the nodes?
I believe I got some SIGBUS in those cases.
> 0006e:
> I'm seeking confirmation, but is this the issue we have discussed
> on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
> $ numactl --membind="0,1" --cpunodebind="0,1"
> /usr/pgsql19/bin/pg_ctl -D /path start
> still shows 4 NUMA nodes used. Current patches use
> numa_num_configured_nodes(), but it says 'This count includes any
> nodes that are currently DISABLED'. So I was wondering if I could help
> by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
> It's the same as You wrote earlier to Alexy?
>
If "mems_allowed" refers to nodes allowing memory allocation, then yes,
this would be one way to get into that issue. Oh, is this what happened
in 0006d?
> > But that's not what you proposed here, clearly. You're saying we should
> > find which NUMA nodes the process is allowed to run, and use those.
> > Instead of just using all *configured* nodes. And I agree with that.
>
> So are you already on it ?
>
>> There are a couple unsolved issues, though. While running the tests, I
>> ran into a bunch of weird issues. I saw two types of failures:
>> 1) Bad address
>> 2) Operation canceled
>
> I did run (with io_uring) a short test(< 10min with -c 128) and didn't
> get those. Could you please share specific tips/workload for
> reproducing this?
>
I did get a couple of "operation canceled" failures, but only on fairly
old kernel versions (6.1 which came as default with the VM). I heard
some suggestions this is a bug in older kernels - I don't have any link
to a bug report / fix, though. But I've been unable to reproduce this on
6.17, so maybe it's true.
For me the failures always happened 10 seconds after the start of the
benchmark (and starting the instance), so it's probably sufficient to
keep the runs ~20 seconds (and maybe restart in between?).
But even then it's fairly rare. I've seen ~10 failures for 500 runs.
I haven't seen more "bad address" cases, I have no idea why. I'm still
guessing it's related to huge pages, so maybe I happened to reserve
enough of them.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-06 14:02 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-11-06 14:02 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On Tue, Nov 4, 2025 at 10:21 PM Tomas Vondra <[email protected]> wrote:
Hi Tomas,
> > 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
> > bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
> > I've thought it is going to be simpler to use to_hex() -- see 0009
> > attached.
> >
>
> I don't know. I added simply because it might be useful for development,
> but we probably don't want to expose these pointers at all.
>
> > 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> > called pg_shm_pgproc?
> >
>
> Right. It does not belong to pg_buffercache at all, I just added it
> there because I've been messing with that code already.
Please keep them in for at least for some time (perhaps standalone
patch marked as not intended to be commited would work?). I find the
view extermely useful as it will allow us pinpointing local-vs-remote
NUMA fetches (we need to know the addres).
> > 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
> > during start:
> >
> > 2025-11-04 10:02:27.055 CET [58464] DEBUG: NUMA:
> > pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
> > num_procs 2523 node 0
> > 2025-11-04 10:02:27.057 CET [58464] DEBUG: NUMA:
> > pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
> > num_procs 2523 node 1
> > 2025-11-04 10:02:27.059 CET [58464] DEBUG: NUMA:
> > pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
> > num_procs 2523 node 2
> > 2025-11-04 10:02:27.061 CET [58464] DEBUG: NUMA:
> > pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
> > num_procs 2523 node 3
> > 2025-11-04 10:02:27.062 CET [58464] DEBUG: NUMA:
> > pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
> > num_procs 38 node -1
> > mbind: Invalid argument
> > mbind: Invalid argument
> > mbind: Invalid argument
> > mbind: Invalid argument
> >
>
> I'll take a look, but I don't recall seeing such errors.
>
Alexy also reported this earlier, here
https://www.postgresql.org/message-id/92e23c85-f646-4bab-b5e0-df30d8ddf4bd%40postgrespro.ru
(just use HP, set some high max_connections). I've double checked this
too , numa_tonode_memory() len needs to HP size.
> > 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
> > initially part of NUMA observability patches but got removed during
> > the course of action), I'm attaching 0008. With that you'll get
> > something a little more up to our standards:
> > 2025-11-04 10:27:07.140 CET [59696] DEBUG:
> > fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
> > 0x7f4f4d4b1660
> > 2025-11-04 10:27:07.140 CET [59696] WARNING: libnuma: ERROR: mbind
> >
>
> Not sure.
Any particular objections? We need to somehow emit them into the logs.
> > 0007f: The "mbind: Invalid argument"" issue itself with the below addition:
[..]
> >
> > but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
> > 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
> > fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
> > (HP) it doesn't complain anymore and I get success:
[..]
>
> Hmm, so it seems like another hugepage-related issue. The mbind manpage
> says this about "len":
>
> EINVAL An invalid value was specified for flags or mode; or addr + len
> was less than addr; or addr is not a multiple of the system page size.
>
> I don't think that requires (addr+len) to be a multiple of page size,
> but maybe that is required.
I do think that 'system page size' means above HP page size, but this
time it's just for fastpath_partition_init(), the earlier one seems to
aligned fine (?? -- i havent really checked but there's no error)
> > 0006d: I've got one SIGBUS during a call to select
> > pg_buffercache_numa_pages(); and it looks like that memory accessed is
> > simply not mapped? (bug)
> >
> > Program received signal SIGBUS, Bus error.
> > pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> > ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> > 386 pg_numa_touch_mem_if_required(ptr);
> > (gdb) print ptr
> > $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
> > (gdb) where
> > #0 pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> > ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> > #1 0x0000561a672a0efe in ExecMakeFunctionResultSet
> > (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
> > argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
> > isDone=isDone@entry=0x561a97e8e5c0) at
> > ../src/backend/executor/execSRF.c:624
> > [..]
> >
> > Postmaster had still attached shm (visible via smaps), and if you
> > compare closely 0x7f4ed0200000 against sorted smaps:
> >
> > 7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
> > /anon_hugepage (deleted)
> > 7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
> > /anon_hugepage (deleted)
> > 7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
> > /anon_hugepage (deleted)
> > 7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
> > /anon_hugepage (deleted)
> > 7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
> > /anon_hugepage (deleted)
> >
> > it's NOT there at all (there's no mmap region starting with
> > 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
> > aware of this new mmaped() regions and instead does simple loop over
> > all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
> > os_page_size)"?
> >
>
> I'm confused. How could that mapping be missing? Was this with huge
> pages / how many did you reserve on the nodes?
OK I made and error and paritally got it correct (it crashes reliably)
and partially mislead You, appologies, let me explain. There were two
questions for me:
a) why we make single mmap() and after numa_tonode_memory() we get
plenty of mappings
b) why we get SIGBUS (I've thought they are not continus, but they are
after triple-checking)
ad a) My testing shows that on HP,as stated initially ("all of this
was on 4s/4 NUMA nodes with HP on"). That's what the codes does, you
get single mmaps() (resulting in single entry in smaps), but afte
noda_tonode_memory() there's many of them. Even on laptop:
System has 1 NUMA nodes (0 to 0).
Attempting to allocate 8.000000 MB of HugeTLB memory...
Successfully allocated HugeTLB memory at 0x755828800000, smaps before:
755828800000-755829000000 rw-s 00000000 00:11 259808
/anon_hugepage (deleted)
Pinning first part (from 0x755828800000) to NUMA node 0...
smaps after:
755828800000-755828c00000 rw-s 00000000 00:11 259808
/anon_hugepage (deleted)
755828c00000-755829000000 rw-s 00400000 00:11 259808
/anon_hugepage (deleted)
Pinning second part (from 0x755828c00000) to NUMA node 0...
smaps after:
755828800000-755828c00000 rw-s 00000000 00:11 259808
/anon_hugepage (deleted)
755828c00000-755829000000 rw-s 00400000 00:11 259808
/anon_hugepage (deleted)
It gets even more funny, below I have 8MB HP=on, but just issue 2x
numa_tonode_memory(for len 2MB on 4MB ptr to node0) (two times for
ptr, second time in half of that):
System has 1 NUMA nodes (0 to 0).
Attempting to allocate 8.000000 MB of HugeTLB memory...
Successfully allocated HugeTLB memory at 0x7302dda00000, smaps before:
7302dda00000-7302de200000 rw-s 00000000 00:11 284859
/anon_hugepage (deleted)
Pinning first part (from 0x7302dda00000) to NUMA node 0...
smaps after:
7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
/anon_hugepage (deleted)
7302ddc00000-7302de200000 rw-s 00200000 00:11 284859
/anon_hugepage (deleted)
Pinning second part (from 0x7302dde00000) to NUMA node 0...
smaps after:
7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
/anon_hugepage (deleted)
7302ddc00000-7302dde00000 rw-s 00200000 00:11 284859
/anon_hugepage (deleted)
7302dde00000-7302de000000 rw-s 00400000 00:11 284859
/anon_hugepage (deleted)
7302de000000-7302de200000 rw-s 00600000 00:11 284859
/anon_hugepage (deleted)
Why 4 instead of 1? Because some mappings are now "default" becauswe
their policy was not altered:
$ grep huge /proc/$(pidof testnumammapsplit)/numa_maps
7302dda00000 bind:0 file=/anon_hugepage\040(deleted) huge
7302ddc00000 default file=/anon_hugepage\040(deleted) huge
7302dde00000 bind:0 file=/anon_hugepage\040(deleted) huge
7302de000000 default file=/anon_hugepage\040(deleted) huge
Back to originnal error, they are consecutive regions and earlier problem is
error: 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
start: 0x7f4921400000
end: 0x7f4f4c000000
so it fits into that range (that was my mistate earlier, using just
grep not checking are they really within that), but...
> Maybe there were not enough huge pages left on one of the nodes?
ad b) right, something like that. I've investigated that SIGBUS there
(it's going to be long):
with shared_buffers=32GB, huge_pages 17715 (+1 from what postgres -C
shared_memory_size_in_huge_pages returns), right after startup, but no
touch:
Program received signal SIGBUS, Bus error.
pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
386 pg_numa_touch_mem_if_required(ptr);
(gdb) where
#0 pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
#1 0x00005571f54ddb7d in ExecMakeTableFunctionResult
(setexpr=0x557203870d40, econtext=0x557203870ba8,
argContext=<optimized out>, expectedDesc=0x557203870f80,
randomAccess=false) at ../src/backend/executor/execSRF.c:234
[..]
(gdb) print ptr
$1 = 0x7f6cf8400000 <error: Cannot access memory at address 0x7f6cf8400000>
(gdb)
then it shows?! no available hugepage on one of the nodes (while gdb
is hanging and preving autorestart):
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free: 299
node1/meminfo:Node 1 HugePages_Free: 299
node2/meminfo:Node 2 HugePages_Free: 299
node3/meminfo:Node 3 HugePages_Free: 0
but they are equal in terms of size:
node0/meminfo:Node 0 HugePages_Total: 4429
node1/meminfo:Node 1 HugePages_Total: 4429
node2/meminfo:Node 2 HugePages_Total: 4429
node3/meminfo:Node 3 HugePages_Total: 4428
smaps shows that this address (7f6cf8400000) is mapped in this mapping:
7f6b49c00000-7f6d49c00000 rw-s 652600000 00:11 86064
/anon_hugepage (deleted)
numa_maps for this region shows this is this mapping on node3 (notice
N3 + bind:3 matches lack of memory on Node 3 HugePAges_Free):
7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
N3=3444 kernelpagesize_kB=2048
the surrounding area of this looks like that:
7f6549c00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=4096
N0=4096 kernelpagesize_kB=2048
7f6749c00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=4096
N1=4096 kernelpagesize_kB=2048
7f6949c00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=4096
N2=4096 kernelpagesize_kB=2048
7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
N3=3444 kernelpagesize_kB=2048 <-- this is the one
7f6d49c00000 default file=/anon_hugepage\040(deleted) huge dirty=107
mapmax=6 N3=107 kernelpagesize_kB=2048
Notice it's just N3=3444, while the others are much larger. So
something was using that hugepages memory on N3:
# grep kernelpagesize_kB=2048 /proc/1679/numa_maps | grep -Po
N[0-4]=[0-9]+ | sort
N0=2
N0=4096
N1=2
N1=4096
N2=2
N2=4096
N3=1
N3=1
N3=1
N3=1
N3=107
N3=13
N3=3
N3=3444
So per above it's not there (at least not as 2MB HP). But the number
of mappings is wild there! (node where it is failing has plenty of
memory, no hugepage memory left, but it has like 40k+ of small
mappings!)
# grep -Po 'N[0-3]=' /proc/1679/numa_maps | sort | uniq -c
17 N0=
10 N1=
3 N2=
40434 N3=
most of them are `anon_inode:[io_uring]` (and I had
max_connections=10k). You may ask why in spite of Andres optimization
for reducing number segments for uring, it's not working for me ? Well
I've just noticed way too silent failure to active this (altough I'm
on 6.14.x):
2025-11-06 13:34:49.128 CET [1658] DEBUG: can't use combined
memory mapping for io_uring, kernel or liburing too old
and I dont have io_uring_queue_init_mem()/HAVE_LIBURING_QUEUE_INIT_MEM
apparently on liburing-2.3 (Debian's default). See [1] for more info
(fix is not commited yet sadly).
Next try, now with io_method = worker and right before start:
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Total
node*/meminfo
node0/meminfo:Node 0 HugePages_Total: 4429
node1/meminfo:Node 1 HugePages_Total: 4429
node2/meminfo:Node 2 HugePages_Total: 4429
node3/meminfo:Node 3 HugePages_Total: 4428
and HugePages_Free were 100% (if postgresql was down). After start
(but without doing anything else):
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free: 4393
node1/meminfo:Node 1 HugePages_Free: 4395
node2/meminfo:Node 2 HugePages_Free: 4395
node3/meminfo:Node 3 HugePages_Free: 3446
So sadly the picture is the same (something stole my HP on N3 and it's
PostgreSQL on it's own). After some time of investigating that ("who
stole my hugepage across whole OS"), I've just added MAP_POPULATE to
the mix of PG_MMAP_FLAGS and got this after start:
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free: 0
node1/meminfo:Node 1 HugePages_Free: 0
node2/meminfo:Node 2 HugePages_Free: 0
node3/meminfo:Node 3 HugePages_Free: 1
and then the SELECT to pg_buffercache_numa works fine(!).
Another ways that I have found to eliminate that SIGBUS
a. Would be to throw much more HugePages (so that node does not run to
HugePages_Free), but that's not real option.
b. Then I've reminded myself that I could be running custom kernel
with experimental CONFIG_READ_ONLY_THP_FOR_FS (to reduce iTLB misses
tranparently with specially linked PG; will double check exact stuff
later), so I've thrown never into
/sys/kernel/mm/transparent_hugepage/enabled and defrag too (yes ,
disabled THP) and with that -- drumroll -- that SELECT works. The very
same PG picture after startup (where earlier it would crash), now
after SELECT it looks like that:
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free: 83
node1/meminfo:Node 1 HugePages_Free: 0
node2/meminfo:Node 2 HugePages_Free: 81
node3/meminfo:Node 3 HugePages_Free: 82
Hope that helps a little. To me it sounds like THP used that memory
somehow and we've also wanted to use. With numa_interleave_ptr() that
wouldn't be a problem because probably it would something else
available, but not here as we indicated exact node.
> > 0006e:
> > I'm seeking confirmation, but is this the issue we have discussed
> > on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
> > $ numactl --membind="0,1" --cpunodebind="0,1"
> > /usr/pgsql19/bin/pg_ctl -D /path start
> > still shows 4 NUMA nodes used. Current patches use
> > numa_num_configured_nodes(), but it says 'This count includes any
> > nodes that are currently DISABLED'. So I was wondering if I could help
> > by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
> > It's the same as You wrote earlier to Alexy?
> >
>
> If "mems_allowed" refers to nodes allowing memory allocation, then yes,
> this would be one way to get into that issue. Oh, is this what happened
> in 0006d?
OK, thanks for confirmation. No, 0006d was about normal numactl run,
without --membind.
> I did get a couple of "operation canceled" failures, but only on fairly
> old kernel versions (6.1 which came as default with the VM).
OK, I'll try to see that later too.
btw QQ regarding partitioned clockwise as I had thought: does this
opens a road towards multiple BGwriters? (outside of this
$thread/v1/PoC)
-J.
[1] - https://www.postgresql.org/message-id/CAKZiRmzxj6Lt1w2ffDoUmN533TgyDeYVULEH1PQFLRyBJSFP6w%40mail.gma...
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-11 11:52 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-11-11 11:52 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
Hi,
here's a rebased patch series, fixing most of the smaller issues from
v20251101, and making cfbot happy (hopefully).
On 11/6/25 15:02, Jakub Wartak wrote:
> On Tue, Nov 4, 2025 at 10:21 PM Tomas Vondra <[email protected]> wrote:
>
> Hi Tomas,
>
>>> 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
>>> bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
>>> I've thought it is going to be simpler to use to_hex() -- see 0009
>>> attached.
>>>
>>
>> I don't know. I added simply because it might be useful for development,
>> but we probably don't want to expose these pointers at all.
>>
>>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
>>> called pg_shm_pgproc?
>>>
>>
>> Right. It does not belong to pg_buffercache at all, I just added it
>> there because I've been messing with that code already.
>
> Please keep them in for at least for some time (perhaps standalone
> patch marked as not intended to be commited would work?). I find the
> view extermely useful as it will allow us pinpointing local-vs-remote
> NUMA fetches (we need to know the addres).
>
Are you referring to the _pgproc view specifically, or also to the view
with buffer partitions? I don't intend to remove the view for shared
buffers, that's indeed useful.
>>> 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
>>> during start:
>>>
>>> 2025-11-04 10:02:27.055 CET [58464] DEBUG: NUMA:
>>> pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
>>> num_procs 2523 node 0
>>> 2025-11-04 10:02:27.057 CET [58464] DEBUG: NUMA:
>>> pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
>>> num_procs 2523 node 1
>>> 2025-11-04 10:02:27.059 CET [58464] DEBUG: NUMA:
>>> pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
>>> num_procs 2523 node 2
>>> 2025-11-04 10:02:27.061 CET [58464] DEBUG: NUMA:
>>> pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
>>> num_procs 2523 node 3
>>> 2025-11-04 10:02:27.062 CET [58464] DEBUG: NUMA:
>>> pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
>>> num_procs 38 node -1
>>> mbind: Invalid argument
>>> mbind: Invalid argument
>>> mbind: Invalid argument
>>> mbind: Invalid argument
>>>
>>
>> I'll take a look, but I don't recall seeing such errors.
>>
>
> Alexy also reported this earlier, here
> https://www.postgresql.org/message-id/92e23c85-f646-4bab-b5e0-df30d8ddf4bd%40postgrespro.ru
> (just use HP, set some high max_connections). I've double checked this
> too , numa_tonode_memory() len needs to HP size.
>
OK, I'll investigate this.
>>> 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
>>> initially part of NUMA observability patches but got removed during
>>> the course of action), I'm attaching 0008. With that you'll get
>>> something a little more up to our standards:
>>> 2025-11-04 10:27:07.140 CET [59696] DEBUG:
>>> fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
>>> 0x7f4f4d4b1660
>>> 2025-11-04 10:27:07.140 CET [59696] WARNING: libnuma: ERROR: mbind
>>>
>>
>> Not sure.
>
> Any particular objections? We need to somehow emit them into the logs.
>
No idea, I think it'd be better to make sure this failure can't happen,
but maybe it's not possible. I don't understand the mbind failure well
enough.
>>> 0007f: The "mbind: Invalid argument"" issue itself with the below addition:
> [..]
>>>
>>> but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
>>> 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
>>> fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
>>> (HP) it doesn't complain anymore and I get success:
> [..]
>>
>> Hmm, so it seems like another hugepage-related issue. The mbind manpage
>> says this about "len":
>>
>> EINVAL An invalid value was specified for flags or mode; or addr + len
>> was less than addr; or addr is not a multiple of the system page size.
>>
>> I don't think that requires (addr+len) to be a multiple of page size,
>> but maybe that is required.
>
> I do think that 'system page size' means above HP page size, but this
> time it's just for fastpath_partition_init(), the earlier one seems to
> aligned fine (?? -- i havent really checked but there's no error)
>
Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
partitioning, I don't think that's likely to go into 19.
>>> 0006d: I've got one SIGBUS during a call to select
>>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
>>> simply not mapped? (bug)
>>>
>>> Program received signal SIGBUS, Bus error.
>>> pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
>>> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>>> 386 pg_numa_touch_mem_if_required(ptr);
>>> (gdb) print ptr
>>> $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
>>> (gdb) where
>>> #0 pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
>>> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>>> #1 0x0000561a672a0efe in ExecMakeFunctionResultSet
>>> (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
>>> argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
>>> isDone=isDone@entry=0x561a97e8e5c0) at
>>> ../src/backend/executor/execSRF.c:624
>>> [..]
>>>
>>> Postmaster had still attached shm (visible via smaps), and if you
>>> compare closely 0x7f4ed0200000 against sorted smaps:
>>>
>>> 7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
>>> /anon_hugepage (deleted)
>>> 7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
>>> /anon_hugepage (deleted)
>>> 7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
>>> /anon_hugepage (deleted)
>>> 7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
>>> /anon_hugepage (deleted)
>>> 7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
>>> /anon_hugepage (deleted)
>>>
>>> it's NOT there at all (there's no mmap region starting with
>>> 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
>>> aware of this new mmaped() regions and instead does simple loop over
>>> all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
>>> os_page_size)"?
>>>
>>
>> I'm confused. How could that mapping be missing? Was this with huge
>> pages / how many did you reserve on the nodes?
>
>
> OK I made and error and paritally got it correct (it crashes reliably)
> and partially mislead You, appologies, let me explain. There were two
> questions for me:
> a) why we make single mmap() and after numa_tonode_memory() we get
> plenty of mappings
> b) why we get SIGBUS (I've thought they are not continus, but they are
> after triple-checking)
>
> ad a) My testing shows that on HP,as stated initially ("all of this
> was on 4s/4 NUMA nodes with HP on"). That's what the codes does, you
> get single mmaps() (resulting in single entry in smaps), but afte
> noda_tonode_memory() there's many of them. Even on laptop:
>
> System has 1 NUMA nodes (0 to 0).
> Attempting to allocate 8.000000 MB of HugeTLB memory...
> Successfully allocated HugeTLB memory at 0x755828800000, smaps before:
> 755828800000-755829000000 rw-s 00000000 00:11 259808
> /anon_hugepage (deleted)
> Pinning first part (from 0x755828800000) to NUMA node 0...
> smaps after:
> 755828800000-755828c00000 rw-s 00000000 00:11 259808
> /anon_hugepage (deleted)
> 755828c00000-755829000000 rw-s 00400000 00:11 259808
> /anon_hugepage (deleted)
> Pinning second part (from 0x755828c00000) to NUMA node 0...
> smaps after:
> 755828800000-755828c00000 rw-s 00000000 00:11 259808
> /anon_hugepage (deleted)
> 755828c00000-755829000000 rw-s 00400000 00:11 259808
> /anon_hugepage (deleted)
>
> It gets even more funny, below I have 8MB HP=on, but just issue 2x
> numa_tonode_memory(for len 2MB on 4MB ptr to node0) (two times for
> ptr, second time in half of that):
>
> System has 1 NUMA nodes (0 to 0).
> Attempting to allocate 8.000000 MB of HugeTLB memory...
> Successfully allocated HugeTLB memory at 0x7302dda00000, smaps before:
> 7302dda00000-7302de200000 rw-s 00000000 00:11 284859
> /anon_hugepage (deleted)
> Pinning first part (from 0x7302dda00000) to NUMA node 0...
> smaps after:
> 7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
> /anon_hugepage (deleted)
> 7302ddc00000-7302de200000 rw-s 00200000 00:11 284859
> /anon_hugepage (deleted)
> Pinning second part (from 0x7302dde00000) to NUMA node 0...
> smaps after:
> 7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
> /anon_hugepage (deleted)
> 7302ddc00000-7302dde00000 rw-s 00200000 00:11 284859
> /anon_hugepage (deleted)
> 7302dde00000-7302de000000 rw-s 00400000 00:11 284859
> /anon_hugepage (deleted)
> 7302de000000-7302de200000 rw-s 00600000 00:11 284859
> /anon_hugepage (deleted)
>
> Why 4 instead of 1? Because some mappings are now "default" becauswe
> their policy was not altered:
>
> $ grep huge /proc/$(pidof testnumammapsplit)/numa_maps
> 7302dda00000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7302ddc00000 default file=/anon_hugepage\040(deleted) huge
> 7302dde00000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7302de000000 default file=/anon_hugepage\040(deleted) huge
>
> Back to originnal error, they are consecutive regions and earlier problem is
>
> error: 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
> start: 0x7f4921400000
> end: 0x7f4f4c000000
>
> so it fits into that range (that was my mistate earlier, using just
> grep not checking are they really within that), but...
>
>> Maybe there were not enough huge pages left on one of the nodes?
>
> ad b) right, something like that. I've investigated that SIGBUS there
> (it's going to be long):
>
> with shared_buffers=32GB, huge_pages 17715 (+1 from what postgres -C
> shared_memory_size_in_huge_pages returns), right after startup, but no
> touch:
>
> Program received signal SIGBUS, Bus error.
> pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> 386 pg_numa_touch_mem_if_required(ptr);
> (gdb) where
> #0 pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> #1 0x00005571f54ddb7d in ExecMakeTableFunctionResult
> (setexpr=0x557203870d40, econtext=0x557203870ba8,
> argContext=<optimized out>, expectedDesc=0x557203870f80,
> randomAccess=false) at ../src/backend/executor/execSRF.c:234
> [..]
> (gdb) print ptr
> $1 = 0x7f6cf8400000 <error: Cannot access memory at address 0x7f6cf8400000>
> (gdb)
>
>
> then it shows?! no available hugepage on one of the nodes (while gdb
> is hanging and preving autorestart):
>
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free: 299
> node1/meminfo:Node 1 HugePages_Free: 299
> node2/meminfo:Node 2 HugePages_Free: 299
> node3/meminfo:Node 3 HugePages_Free: 0
>
> but they are equal in terms of size:
> node0/meminfo:Node 0 HugePages_Total: 4429
> node1/meminfo:Node 1 HugePages_Total: 4429
> node2/meminfo:Node 2 HugePages_Total: 4429
> node3/meminfo:Node 3 HugePages_Total: 4428
>
> smaps shows that this address (7f6cf8400000) is mapped in this mapping:
> 7f6b49c00000-7f6d49c00000 rw-s 652600000 00:11 86064
> /anon_hugepage (deleted)
>
> numa_maps for this region shows this is this mapping on node3 (notice
> N3 + bind:3 matches lack of memory on Node 3 HugePAges_Free):
> 7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
> N3=3444 kernelpagesize_kB=2048
>
> the surrounding area of this looks like that:
>
> 7f6549c00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=4096
> N0=4096 kernelpagesize_kB=2048
> 7f6749c00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=4096
> N1=4096 kernelpagesize_kB=2048
> 7f6949c00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=4096
> N2=4096 kernelpagesize_kB=2048
> 7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
> N3=3444 kernelpagesize_kB=2048 <-- this is the one
> 7f6d49c00000 default file=/anon_hugepage\040(deleted) huge dirty=107
> mapmax=6 N3=107 kernelpagesize_kB=2048
>
> Notice it's just N3=3444, while the others are much larger. So
> something was using that hugepages memory on N3:
>
> # grep kernelpagesize_kB=2048 /proc/1679/numa_maps | grep -Po
> N[0-4]=[0-9]+ | sort
> N0=2
> N0=4096
> N1=2
> N1=4096
> N2=2
> N2=4096
> N3=1
> N3=1
> N3=1
> N3=1
> N3=107
> N3=13
> N3=3
> N3=3444
>
> So per above it's not there (at least not as 2MB HP). But the number
> of mappings is wild there! (node where it is failing has plenty of
> memory, no hugepage memory left, but it has like 40k+ of small
> mappings!)
>
> # grep -Po 'N[0-3]=' /proc/1679/numa_maps | sort | uniq -c
> 17 N0=
> 10 N1=
> 3 N2=
> 40434 N3=
>
> most of them are `anon_inode:[io_uring]` (and I had
> max_connections=10k). You may ask why in spite of Andres optimization
> for reducing number segments for uring, it's not working for me ? Well
> I've just noticed way too silent failure to active this (altough I'm
> on 6.14.x):
> 2025-11-06 13:34:49.128 CET [1658] DEBUG: can't use combined
> memory mapping for io_uring, kernel or liburing too old
> and I dont have io_uring_queue_init_mem()/HAVE_LIBURING_QUEUE_INIT_MEM
> apparently on liburing-2.3 (Debian's default). See [1] for more info
> (fix is not commited yet sadly).
>
> Next try, now with io_method = worker and right before start:
>
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Total
> node*/meminfo
> node0/meminfo:Node 0 HugePages_Total: 4429
> node1/meminfo:Node 1 HugePages_Total: 4429
> node2/meminfo:Node 2 HugePages_Total: 4429
> node3/meminfo:Node 3 HugePages_Total: 4428
> and HugePages_Free were 100% (if postgresql was down). After start
> (but without doing anything else):
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free: 4393
> node1/meminfo:Node 1 HugePages_Free: 4395
> node2/meminfo:Node 2 HugePages_Free: 4395
> node3/meminfo:Node 3 HugePages_Free: 3446
>
> So sadly the picture is the same (something stole my HP on N3 and it's
> PostgreSQL on it's own). After some time of investigating that ("who
> stole my hugepage across whole OS"), I've just added MAP_POPULATE to
> the mix of PG_MMAP_FLAGS and got this after start:
>
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free: 0
> node1/meminfo:Node 1 HugePages_Free: 0
> node2/meminfo:Node 2 HugePages_Free: 0
> node3/meminfo:Node 3 HugePages_Free: 1
>
> and then the SELECT to pg_buffercache_numa works fine(!).
>
> Another ways that I have found to eliminate that SIGBUS
> a. Would be to throw much more HugePages (so that node does not run to
> HugePages_Free), but that's not real option.
> b. Then I've reminded myself that I could be running custom kernel
> with experimental CONFIG_READ_ONLY_THP_FOR_FS (to reduce iTLB misses
> tranparently with specially linked PG; will double check exact stuff
> later), so I've thrown never into
> /sys/kernel/mm/transparent_hugepage/enabled and defrag too (yes ,
> disabled THP) and with that -- drumroll -- that SELECT works. The very
> same PG picture after startup (where earlier it would crash), now
> after SELECT it looks like that:
>
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free: 83
> node1/meminfo:Node 1 HugePages_Free: 0
> node2/meminfo:Node 2 HugePages_Free: 81
> node3/meminfo:Node 3 HugePages_Free: 82
>
> Hope that helps a little. To me it sounds like THP used that memory
> somehow and we've also wanted to use. With numa_interleave_ptr() that
> wouldn't be a problem because probably it would something else
> available, but not here as we indicated exact node.
>
>>> 0006e:
>>> I'm seeking confirmation, but is this the issue we have discussed
>>> on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
>>> $ numactl --membind="0,1" --cpunodebind="0,1"
>>> /usr/pgsql19/bin/pg_ctl -D /path start
>>> still shows 4 NUMA nodes used. Current patches use
>>> numa_num_configured_nodes(), but it says 'This count includes any
>>> nodes that are currently DISABLED'. So I was wondering if I could help
>>> by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
>>> It's the same as You wrote earlier to Alexy?
>>>
>>
>> If "mems_allowed" refers to nodes allowing memory allocation, then yes,
>> this would be one way to get into that issue. Oh, is this what happened
>> in 0006d?
>
> OK, thanks for confirmation. No, 0006d was about normal numactl run,
> without --membind.
>
I didn't have time to look into all this info about mappings, io_uring
yet, so no response from me.
>> I did get a couple of "operation canceled" failures, but only on fairly
>> old kernel versions (6.1 which came as default with the VM).
>
> OK, I'll try to see that later too.
>
> btw QQ regarding partitioned clockwise as I had thought: does this
> opens a road towards multiple BGwriters? (outside of this
> $thread/v1/PoC)
>
I don't think the clocksweep partitioning is required for multiple
bgwriters, but it might make it easier.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20251111-0007-NUMA-partition-PGPROC.patch (49.2K, ../../[email protected]/2-v20251111-0007-NUMA-partition-PGPROC.patch)
download | inline diff:
From 7f8096d294ed9d69c953ff97be7f46db058cd2cb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:10:03 +0100
Subject: [PATCH v20251111 7/7] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 96 ++-
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 ++-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 551 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 723 insertions(+), 73 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 1379d54cc5d..f340fcf420c 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -29,7 +30,8 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -931,3 +934,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index ea43b432daf..7d589bac115 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -575,7 +575,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -634,7 +634,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 89d0bfa7760..e0e17293536 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..3288900bb6f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -292,7 +292,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..f5844aa5b6a 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 587859a5754..bdcdbcc6b5f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -799,6 +799,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 810a549efce..0937292643f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -472,7 +472,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 4cc7f645c31..d01f486876d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..0a0ce98b725 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,22 +29,33 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "access/xlogwait.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -77,8 +88,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -91,6 +102,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -101,11 +135,41 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -130,6 +194,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -141,6 +259,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -149,6 +270,8 @@ ProcGlobalShmemSize(void)
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
+
return size;
}
@@ -193,7 +316,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -212,6 +335,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -226,6 +352,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,21 +376,110 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -291,23 +515,91 @@ InitProcGlobal(void)
/* Reserve space for semaphores. */
PGReserveSemaphores(ProcGlobalSemas());
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
- /* Common initialization for all PGPROCs, regardless of type. */
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -371,9 +663,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -440,7 +729,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -651,7 +984,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1059,7 +1392,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1108,7 +1441,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1998,7 +2331,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2073,3 +2406,173 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ *
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_partition_init procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cb52c417592..b11b04818be 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1881,6 +1881,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.1
[text/x-patch] v20251111-0006-NUMA-shared-buffers-partitioning.patch (44.0K, ../../[email protected]/3-v20251111-0006-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 705ce91522059afc98ff339cc640ab7aa38d4ac7 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:05:35 +0100
Subject: [PATCH v20251111 6/7] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
.../pg_buffercache--1.6--1.7.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 44 +-
src/backend/storage/buffer/buf_init.c | 569 +++++++++++++++++-
src/backend/storage/buffer/freelist.c | 88 ++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 14 +-
src/include/storage/bufmgr.h | 4 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
11 files changed, 736 insertions(+), 68 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 2c4d560514d..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -13,6 +13,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer, -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8536e2debef..1379d54cc5d 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -813,19 +813,21 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
INT8OID, -1, 0);
@@ -849,7 +851,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -868,7 +871,7 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
@@ -886,36 +889,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
- values[4] = Int64GetDatum(complete_passes);
+ values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
- values[5] = Int32GetDatum(next_victim_buffer);
+ values[5] = Int64GetDatum(complete_passes);
nulls[5] = false;
- values[6] = Int64GetDatum(buffer_total_allocs);
+ values[6] = Int32GetDatum(next_victim_buffer);
nulls[6] = false;
- values[7] = Int64GetDatum(buffer_allocs);
+ values[7] = Int64GetDatum(buffer_total_allocs);
nulls[7] = false;
- values[8] = Int64GetDatum(buffer_total_req_allocs);
+ values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
- values[9] = Int64GetDatum(buffer_req_allocs);
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
nulls[9] = false;
- values[10] = PointerGetDatum(array);
+ values[10] = Int64GetDatum(buffer_req_allocs);
nulls[10] = false;
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 0362fda24aa..587859a5754 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,12 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -29,15 +35,24 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
-/* *
- * number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
/* Array of structs with information about buffer ranges */
BufferPartitions *BufferPartitionsArray = NULL;
+static void buffer_partitions_prepare(void);
static void buffer_partitions_init(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -85,25 +100,85 @@ BufferManagerShmemInit(void)
foundIOCV,
foundBufCkpt,
foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
/* allocate the partition registry first */
BufferPartitionsArray = (BufferPartitions *)
ShmemInitStruct("Buffer Partitions",
offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ mul_size(sizeof(BufferPartition), numa_partitions),
&foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -133,7 +208,10 @@ BufferManagerShmemInit(void)
{
int i;
- /* Initialize buffer partitions (calculate buffer ranges). */
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
buffer_partitions_init();
/*
@@ -172,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -201,11 +286,244 @@ BufferManagerShmemSize(void)
/* account for registry of NUMA partitions */
size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+ mul_size(sizeof(BufferPartition), numa_partitions)));
return size;
}
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches earlier in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
/*
* Sanity checks of buffers partitions - there must be no gaps, it must cover
* the whole range of buffers, etc.
@@ -267,33 +585,137 @@ buffer_partitions_init(void)
{
int remaining_buffers = NBuffers;
int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers
- = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
-
- BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
- for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[n];
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
- /* buffers this partition should get (last partition can get fewer) */
- int num_buffers = Min(remaining_buffers, part_buffers);
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
- remaining_buffers -= num_buffers;
+ remaining_buffers -= node_buffers;
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- buffer += num_buffers;
+ /* XXX we should get the actual node ID from the mask */
+ if (numa_can_partition)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
}
AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ char *startptr,
+ *endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
}
int
@@ -302,14 +724,21 @@ BufferPartitionCount(void)
return BufferPartitionsArray->npartitions;
}
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
{
BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -322,8 +751,82 @@ BufferPartitionGet(int idx, int *num_buffers,
/* return parameters before the partitions are initialized (during sizing) */
void
-BufferPartitionParams(int *num_partitions)
+BufferPartitionParams(int *num_partitions, int *num_nodes)
{
if (num_partitions)
- *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 8be77a9c8b1..810a549efce 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -124,7 +124,9 @@ typedef struct
//int __attribute__((aligned(64))) bgwprocno;
/* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
int num_partitions;
+ int num_partitions_per_node;
/* clocksweep partitions */
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
@@ -270,16 +272,72 @@ ClockSweepTick(ClockSweep *sweep)
* calculate_partition_index
* calculate the buffer / clock-sweep partition to use
*
- * use PID to determine the buffer partition
- *
- * XXX We could use NUMA node / core ID to pick partition, but we'd need
- * to handle cases with fewer nodes/cores than partitions somehow. Although,
- * maybe the balancing would handle that too.
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
*/
static int
calculate_partition_index(void)
{
- return (MyProcPid % StrategyControl->num_partitions);
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
}
/*
@@ -947,7 +1005,7 @@ StrategyShmemSize(void)
Size size = 0;
int num_partitions;
- BufferPartitionParams(&num_partitions);
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -974,9 +1032,17 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
int num_partitions;
+ int num_partitions_per_node;
num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -1011,7 +1077,8 @@ StrategyInitialize(bool init)
/* Initialize the clock sweep pointers (for all partitions) */
for (int i = 0; i < num_partitions; i++)
{
- int num_buffers,
+ int node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -1020,7 +1087,8 @@ StrategyInitialize(bool init)
pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
/* get info about the buffer partition */
- BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
/*
* FIXME This may not quite right, because if NBuffers is not a
@@ -1056,6 +1124,8 @@ StrategyInitialize(bool init)
/* initialize the partitioned clocksweep */
StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 1128167c025..8192c27066b 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -636,6 +636,16 @@
options => 'debug_logical_replication_streaming_options',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'debug_parallel_query', type => 'enum', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Forces the planner\'s use parallel query nodes.',
long_desc => 'This can be useful for testing the parallel query infrastructure by forcing the planner to generate plans that contain nodes that perform tuple communication between workers and the main process.',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0209b2067a2..404eb3432f9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1118b386228..33377841c57 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -299,10 +299,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -312,7 +312,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -555,8 +555,8 @@ extern void AtEOXact_LocalBuffers(bool isCommit);
extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
-extern void BufferPartitionParams(int *num_partitions);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4e7b1fcd4ab..510018db115 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,10 +156,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -169,6 +171,7 @@ typedef struct BufferPartition
typedef struct BufferPartitions
{
int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -346,6 +349,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 3368a43a338..8ee0e7d211c 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -106,6 +109,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -128,4 +161,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
--
2.51.1
[text/x-patch] v20251111-0005-clock-sweep-weighted-balancing.patch (5.2K, ../../[email protected]/4-v20251111-0005-clock-sweep-weighted-balancing.patch)
download | inline diff:
From c97635bdd35858b44de03012268f521678cb1edc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251111 5/7] clock-sweep: weighted balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
Note: This may be more important with NUMA-aware partitioning, which
restricts the allowed sizes of partiions (especially with huge pages).
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 3af82e267c6..8be77a9c8b1 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -619,6 +619,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -645,16 +659,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -662,8 +687,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -726,6 +757,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -733,7 +768,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -747,22 +782,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.1
[text/x-patch] v20251111-0004-clock-sweep-scan-all-partitions.patch (6.7K, ../../[email protected]/5-v20251111-0004-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From be684d592380765e857bcc85fb32807c5212e8b2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251111 4/7] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 169071032b4..3af82e267c6 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -167,6 +167,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -201,10 +204,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -370,7 +372,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -424,37 +427,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -482,7 +517,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -501,7 +536,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.1
[text/x-patch] v20251111-0003-clock-sweep-balancing-of-allocations.patch (25.3K, ../../[email protected]/6-v20251111-0003-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From 39316faa1c8bd0f20320b4481cb0c05bab14dd46 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:45:34 +0100
Subject: [PATCH v20251111 3/7] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 14e750beeff..2c4d560514d 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -21,7 +21,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 1856efb8786..8536e2debef 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -794,6 +796,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -823,6 +827,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,11 +853,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -856,8 +872,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -883,6 +907,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffer_allocs);
nulls[7] = false;
+ values[8] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[8] = false;
+
+ values[9] = Int64GetDatum(buffer_req_allocs);
+ nulls[9] = false;
+
+ values[10] = PointerGetDatum(array);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a3092ce801d..82c645a3b00 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3912,6 +3912,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d40b09f7e69..169071032b4 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -130,7 +166,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -142,7 +204,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -233,11 +295,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -309,7 +419,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -417,6 +527,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -443,6 +771,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -627,7 +956,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1001,8 +1344,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1010,11 +1355,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3307190f611..1118b386228 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,6 +508,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7052f9de57c..4e7b1fcd4ab 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -360,11 +360,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.1
[text/x-patch] v20251111-0002-clock-sweep-basic-partitioning.patch (33.9K, ../../[email protected]/7-v20251111-0002-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 34c5d3574fa7efa48de7d5a99921911b63d04b30 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:03:32 +0100
Subject: [PATCH v20251111 2/7] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 38 ++-
src/backend/storage/buffer/buf_init.c | 8 +
src/backend/storage/buffer/bufmgr.c | 186 ++++++++----
src/backend/storage/buffer/freelist.c | 283 +++++++++++++++---
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 433 insertions(+), 106 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index f1c20960b7e..14e750beeff 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -15,7 +15,13 @@ CREATE VIEW pg_buffercache_partitions AS
(partition integer, -- partition index
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8a980bd1864..1856efb8786 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -809,12 +809,20 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 1, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -835,12 +843,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -853,6 +871,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[3] = Int32GetDatum(last_buffer);
nulls[3] = false;
+ values[4] = Int64GetDatum(complete_passes);
+ nulls[4] = false;
+
+ values[5] = Int32GetDatum(next_victim_buffer);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffer_total_allocs);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_allocs);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 528a368a8b7..0362fda24aa 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -319,3 +319,11 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions)
+{
+ if (num_partitions)
+ *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 327ddb7adc8..a3092ce801d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3608,33 +3608,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3662,25 +3658,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3692,17 +3679,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3710,11 +3697,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3734,9 +3721,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3750,15 +3737,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3766,9 +3754,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3778,7 +3766,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3786,10 +3774,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3816,7 +3804,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3841,20 +3829,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3868,7 +3856,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3898,8 +3886,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 28d952b3534..d40b09f7e69 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +130,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +142,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +150,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +183,61 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * use PID to determine the buffer partition
+ *
+ * XXX We could use NUMA node / core ID to pick partition, but we'd need
+ * to handle cases with fewer nodes/cores than partitions somehow. Although,
+ * maybe the balancing would handle that too.
+ */
+static int
+calculate_partition_index(void)
+{
+ return (MyProcPid % StrategyControl->num_partitions);
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +309,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +417,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +464,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +538,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +548,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +567,10 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_partitions;
+
+ num_partitions = BufferPartitionCount();
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +588,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +601,40 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
}
else
Assert(!init);
@@ -803,3 +998,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 139055a4a7d..3307190f611 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,7 +508,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -554,5 +556,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 24860c6c2c4..7052f9de57c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -359,6 +359,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 56e25026fbf..cb52c417592 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.1
[text/x-patch] v20251111-0001-Infrastructure-for-partitioning-shared-buf.patch (14.9K, ../../[email protected]/8-v20251111-0001-Infrastructure-for-partitioning-shared-buf.patch)
download | inline diff:
From fc35b864c5350b8640000368ddc1d82bd4ad3756 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251111 1/7] Infrastructure for partitioning shared buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep).
The registry is a small BufferPartitions array in shared memory, with
partitions sized to be a fair share of shared buffers. Later patches may
improve this to consider NUMA, and similar details.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 25 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 144 +++++++++++++++++-
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
9 files changed, 284 insertions(+), 3 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..f1c20960b7e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,25 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ab790533ff6..8a980bd1864 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -776,3 +778,87 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..528a368a8b7 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -17,6 +17,11 @@
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +29,14 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/* *
+ * number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_init(void);
/*
* Data Structures:
@@ -70,7 +83,15 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ &foundParts);
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -112,6 +133,9 @@ BufferManagerShmemInit(void)
{
int i;
+ /* Initialize buffer partitions (calculate buffer ranges). */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -175,5 +199,123 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+
return size;
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers
+ = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[n];
+
+ /* buffers this partition should get (last partition can get fewer) */
+ int num_buffers = Min(remaining_buffers, part_buffers);
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5400c56a965..139055a4a7d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -345,6 +345,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -549,4 +550,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b5f8f3c5d42..24860c6c2c4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -153,6 +153,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 432509277c9..56e25026fbf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-17 09:23 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-11-17 09:23 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On Tue, Nov 11, 2025 at 12:52 PM Tomas Vondra <[email protected]> wrote:
>
> Hi,
>
> here's a rebased patch series, fixing most of the smaller issues from
> v20251101, and making cfbot happy (hopefully).
Hi Tomas,
> >>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> >>> called pg_shm_pgproc?
> >>>
> >>
> >> Right. It does not belong to pg_buffercache at all, I just added it
> >> there because I've been messing with that code already.
> >
> > Please keep them in for at least for some time (perhaps standalone
> > patch marked as not intended to be commited would work?). I find the
> > view extermely useful as it will allow us pinpointing local-vs-remote
> > NUMA fetches (we need to know the addres).
> >
>
> Are you referring to the _pgproc view specifically, or also to the view
> with buffer partitions? I don't intend to remove the view for shared
> buffers, that's indeed useful.
Both, even the _pgproc.
> Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
> partitioning, I don't think that's likely to go into 19.
Oh ok.
> >>> 0006d: I've got one SIGBUS during a call to select
> >>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
> >>> simply not mapped? (bug)
[..]
> I didn't have time to look into all this info about mappings, io_uring
> yet, so no response from me.
>
Ok, so the proper HP + SIGBUS explanation:
Appologies, earlier I wrote that disabling THP does workaround this,
but I've probably made an error there and used wrong binary back there
(with MAP_POPULATE in PG_MMAP_FLAGS), so please ignore that.
1. Before starting PG, with shared_buffers=32GB, huge_pages=on (2MB
ones), vm.nr_hugepages=17715, 4 NUMA nodes, kernel 6.14.x,
max_connections=10k, wal_buffers=1GB:
node0/hugepages/hugepages-2048kB/free_hugepages:4429
node1/hugepages/hugepages-2048kB/free_hugepages:4429
node2/hugepages/hugepages-2048kB/free_hugepages:4429
node3/hugepages/hugepages-2048kB/free_hugepages:4428
2. Just startup the PG with the older NUMA patchset 20251101. There
will be deficit across NUMA nodes right after startup, mostly one node
NUMA will allocate much more:
node0/hugepages/hugepages-2048kB/free_hugepages:4397
node1/hugepages/hugepages-2048kB/free_hugepages:3453
node2/hugepages/hugepages-2048kB/free_hugepages:4397
node3/hugepages/hugepages-2048kB/free_hugepages:4396
3. Check layout of NUMA maps for postmaster PID
7fc9cb200000 default file=/anon_hugepage\040(deleted) huge dirty=517
mapmax=8 N1=517 kernelpagesize_kB=2048 [!!!]
7fca0d600000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N0=32 kernelpagesize_kB=2048
7fca11600000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N1=32 kernelpagesize_kB=2048
7fca15600000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N2=32 kernelpagesize_kB=2048
7fca19600000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N3=32 kernelpagesize_kB=2048
7fca1d600000 default file=/anon_hugepage\040(deleted) huge
7fca1d800000 bind:0 file=/anon_hugepage\040(deleted) huge
7fcc1d800000 bind:1 file=/anon_hugepage\040(deleted) huge
7fce1d800000 bind:2 file=/anon_hugepage\040(deleted) huge
7fd01d800000 bind:3 file=/anon_hugepage\040(deleted) huge
7fd21d800000 default file=/anon_hugepage\040(deleted) huge dirty=425
mapmax=8 N1=425 kernelpagesize_kB=2048 [!!!]
So your patch doesn't do anything special for anything other than
Buffer Blocks and PGPROC in the above picture, so the the default
mmap() just keeps on with "default" NUMA policy which takes per above
(517+425) * 2MB = ~1884 MB of really used memory as per N1 entires. PG
does touch those regions on startup, but it doesnt really touch Buffer
Blocks. Anyway, this causes the missing amount of free huge pages on
the N1 (generates pressure on this Node 1).
So as it stands, the patchset is missing some form balancing to use
equal memory across nodes:
- each node to be forced to get certain amount of BufferBlocks/NUMA nodes blocks
- yet we do nothing and leave at the "defaults" the others regions
(e..g $SegHDR (start of shm) .. first Buffers Block), as those are
placed on the current node (due default policy), which in causes turns
this memory overallocation imbalance (so in the example N1 will get
Buffer Blocks + everything else, but that only happens on real access
not during mmap() due to lazy/first touch policy)
Currently, any launch of anything that touches imbalanced NUMA node
memory with deficit (N1 above) - use of pg_shm_allocations,
pg_buffercache - it will cause stress there and end up in SIGBUS.
This looks by design on Linux kernel side: exc:page_fault() ->
do_user_addr_fault() -> do_sigbus() AKA force_sig_fault(). But, if I
hack pg to hack do interleave (or just numactl --interleave=all ... )
to effectivley interleave those 3 "default" regions instead, so I'll
get "interleave" like that:
7fb2dd000000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
dirty=517 mapmax=8 N0=129 N1=132 N2=128 N3=128 kernelpagesize_kB=2048
7fb31f400000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N0=32 kernelpagesize_kB=2048
7fb323400000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N1=32 kernelpagesize_kB=2048
7fb327400000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N2=32 kernelpagesize_kB=2048
7fb32b400000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N3=32 kernelpagesize_kB=2048
7fb32f400000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
7fb32f600000 bind:0 file=/anon_hugepage\040(deleted) huge
7fb52f600000 bind:1 file=/anon_hugepage\040(deleted) huge
7fb72f600000 bind:2 file=/anon_hugepage\040(deleted) huge
7fb92f600000 bind:3 file=/anon_hugepage\040(deleted) huge
7fbb2f600000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
dirty=425 N0=106 N1=106 N2=105 N3=108 kernelpagesize_kB=2048
then even after fully touching everything (via select to
pg_shm_allocations), it'll run, I'll get much better balance, and wont
have SIGBUS issues:
node0/hugepages/hugepages-2048kB/free_hugepages:23
node1/hugepages/hugepages-2048kB/free_hugepages:23
node2/hugepages/hugepages-2048kB/free_hugepages:23
node3/hugepages/hugepages-2048kB/free_hugepages:22
This somehow demonstrates that enough free memory is out there, it's
just imbalance that causes SIGBUS. I hope this somehow hopefully
answers one of Your's main questions as per in the very first messages
what we should do with remaining shared_buffer members. I would like
to hear your thoughts on this, before I start benchmarking this for
real as I didnt want to bench it yet, as such interleaving could alter
the the test results.
Other things I've noticed:
- smaps Size: && Shared_Hugetlb: reporting are a lie and are showing
really touched memory, not assigned memory
- same goes for procfs's numa_maps, ignore the N[0-3] sizes, it's only
"really used", not assigned
- the best is just to manually calculate size from pointers/address range itself
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-17 17:28 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-11-17 17:28 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On 11/17/25 10:23, Jakub Wartak wrote:
> On Tue, Nov 11, 2025 at 12:52 PM Tomas Vondra <[email protected]> wrote:
>>
>> Hi,
>>
>> here's a rebased patch series, fixing most of the smaller issues from
>> v20251101, and making cfbot happy (hopefully).
>
> Hi Tomas,
>
>>>>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
>>>>> called pg_shm_pgproc?
>>>>>
>>>>
>>>> Right. It does not belong to pg_buffercache at all, I just added it
>>>> there because I've been messing with that code already.
>>>
>>> Please keep them in for at least for some time (perhaps standalone
>>> patch marked as not intended to be commited would work?). I find the
>>> view extermely useful as it will allow us pinpointing local-vs-remote
>>> NUMA fetches (we need to know the addres).
>>>
>>
>> Are you referring to the _pgproc view specifically, or also to the view
>> with buffer partitions? I don't intend to remove the view for shared
>> buffers, that's indeed useful.
>
> Both, even the _pgproc.
>
>
>> Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
>> partitioning, I don't think that's likely to go into 19.
>
> Oh ok.
>
>>>>> 0006d: I've got one SIGBUS during a call to select
>>>>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
>>>>> simply not mapped? (bug)
> [..]
>> I didn't have time to look into all this info about mappings, io_uring
>> yet, so no response from me.
>>
>
> Ok, so the proper HP + SIGBUS explanation:
>
> Appologies, earlier I wrote that disabling THP does workaround this,
> but I've probably made an error there and used wrong binary back there
> (with MAP_POPULATE in PG_MMAP_FLAGS), so please ignore that.
>
> 1. Before starting PG, with shared_buffers=32GB, huge_pages=on (2MB
> ones), vm.nr_hugepages=17715, 4 NUMA nodes, kernel 6.14.x,
> max_connections=10k, wal_buffers=1GB:
>
> node0/hugepages/hugepages-2048kB/free_hugepages:4429
> node1/hugepages/hugepages-2048kB/free_hugepages:4429
> node2/hugepages/hugepages-2048kB/free_hugepages:4429
> node3/hugepages/hugepages-2048kB/free_hugepages:4428
>
> 2. Just startup the PG with the older NUMA patchset 20251101. There
> will be deficit across NUMA nodes right after startup, mostly one node
> NUMA will allocate much more:
>
> node0/hugepages/hugepages-2048kB/free_hugepages:4397
> node1/hugepages/hugepages-2048kB/free_hugepages:3453
> node2/hugepages/hugepages-2048kB/free_hugepages:4397
> node3/hugepages/hugepages-2048kB/free_hugepages:4396
>
> 3. Check layout of NUMA maps for postmaster PID
>
> 7fc9cb200000 default file=/anon_hugepage\040(deleted) huge dirty=517
> mapmax=8 N1=517 kernelpagesize_kB=2048 [!!!]
> 7fca0d600000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N0=32 kernelpagesize_kB=2048
> 7fca11600000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N1=32 kernelpagesize_kB=2048
> 7fca15600000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N2=32 kernelpagesize_kB=2048
> 7fca19600000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N3=32 kernelpagesize_kB=2048
> 7fca1d600000 default file=/anon_hugepage\040(deleted) huge
> 7fca1d800000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7fcc1d800000 bind:1 file=/anon_hugepage\040(deleted) huge
> 7fce1d800000 bind:2 file=/anon_hugepage\040(deleted) huge
> 7fd01d800000 bind:3 file=/anon_hugepage\040(deleted) huge
> 7fd21d800000 default file=/anon_hugepage\040(deleted) huge dirty=425
> mapmax=8 N1=425 kernelpagesize_kB=2048 [!!!]
>
> So your patch doesn't do anything special for anything other than
> Buffer Blocks and PGPROC in the above picture, so the the default
> mmap() just keeps on with "default" NUMA policy which takes per above
> (517+425) * 2MB = ~1884 MB of really used memory as per N1 entires. PG
> does touch those regions on startup, but it doesnt really touch Buffer
> Blocks. Anyway, this causes the missing amount of free huge pages on
> the N1 (generates pressure on this Node 1).
>
> So as it stands, the patchset is missing some form balancing to use
> equal memory across nodes:
> - each node to be forced to get certain amount of BufferBlocks/NUMA nodes blocks
> - yet we do nothing and leave at the "defaults" the others regions
> (e..g $SegHDR (start of shm) .. first Buffers Block), as those are
> placed on the current node (due default policy), which in causes turns
> this memory overallocation imbalance (so in the example N1 will get
> Buffer Blocks + everything else, but that only happens on real access
> not during mmap() due to lazy/first touch policy)
>
> Currently, any launch of anything that touches imbalanced NUMA node
> memory with deficit (N1 above) - use of pg_shm_allocations,
> pg_buffercache - it will cause stress there and end up in SIGBUS.
> This looks by design on Linux kernel side: exc:page_fault() ->
> do_user_addr_fault() -> do_sigbus() AKA force_sig_fault(). But, if I
> hack pg to hack do interleave (or just numactl --interleave=all ... )
> to effectivley interleave those 3 "default" regions instead, so I'll
> get "interleave" like that:
>
> 7fb2dd000000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> dirty=517 mapmax=8 N0=129 N1=132 N2=128 N3=128 kernelpagesize_kB=2048
> 7fb31f400000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N0=32 kernelpagesize_kB=2048
> 7fb323400000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N1=32 kernelpagesize_kB=2048
> 7fb327400000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N2=32 kernelpagesize_kB=2048
> 7fb32b400000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N3=32 kernelpagesize_kB=2048
> 7fb32f400000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> 7fb32f600000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7fb52f600000 bind:1 file=/anon_hugepage\040(deleted) huge
> 7fb72f600000 bind:2 file=/anon_hugepage\040(deleted) huge
> 7fb92f600000 bind:3 file=/anon_hugepage\040(deleted) huge
> 7fbb2f600000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> dirty=425 N0=106 N1=106 N2=105 N3=108 kernelpagesize_kB=2048
>
> then even after fully touching everything (via select to
> pg_shm_allocations), it'll run, I'll get much better balance, and wont
> have SIGBUS issues:
>
> node0/hugepages/hugepages-2048kB/free_hugepages:23
> node1/hugepages/hugepages-2048kB/free_hugepages:23
> node2/hugepages/hugepages-2048kB/free_hugepages:23
> node3/hugepages/hugepages-2048kB/free_hugepages:22
>
> This somehow demonstrates that enough free memory is out there, it's
> just imbalance that causes SIGBUS. I hope this somehow hopefully
> answers one of Your's main questions as per in the very first messages
> what we should do with remaining shared_buffer members. I would like
> to hear your thoughts on this, before I start benchmarking this for
> real as I didnt want to bench it yet, as such interleaving could alter
> the the test results.
>
Thanks for investigating this. If I understand the findings correctly,
it agrees with my imprecise explanation in [1], right? There I said:
> ...
> You may ask why the per-node limit is too low. We still need just
> shared_memory_size_in_huge_pages, right? And if we were partitioning
> the whole memory segment, that'd be true. But we only to that for
> shared buffers, and there's a lot of other shared memory - could be
> 1-2GB or so, depending on the configuration.
>
> And this gets placed on one of the nodes, and it counts against the
> limit on that particular node. And so it doesn't have enough huge
> pages to back the partition of shared buffers.
> ...
Which I think is mostly the same thing you're saying, and you have the
maps to support it.
In any case, I think setting "interleave" as the default policy, and
then overriding it for the areas we partition explicitly (buffers,
pgproc), seems like the right solution. The only other solution would be
balance it ourselves, but how is that different from interleaving?
So I think this makes sense, and you can do --interleave=all for the
benchmark.
[1]
https://www.postgresql.org/message-id/71a46484-053c-4b81-ba32-ddac050a8b5d%40vondra.me
I suppose we may need to adjust shared_memory_size_in_huge_pages,
because the interleave followed by explicit partitioning may still leave
behind a bit of imbalance. It should be only a couple pages, but I
haven't done the math yet.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-21 18:49 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-11-21 18:49 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
Hi,
Here's an updated version of the patch series.
It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
and a incorrect array length.
The main change is in 0006 - it sets the default allocation policy for
shmem to interleaving, before doing the explicit partitioning for shared
buffers. It does it by calling numa_set_membind before the mmap(), and
then numa_interleave_memory() on the allocated shmem. It does this to
allow using MAP_POPULATE - but that's commented out by default.
This does seem to solve the SIGBUS failures for me. I still think there
might be a small chance of hitting that, because of locating an extra
"boundary" page on one of the nodes. But it should be solvable by
reserving a couple more pages.
Jakub, what do you think?
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20251121-0007-NUMA-partition-PGPROC.patch (49.2K, ../../[email protected]/2-v20251121-0007-NUMA-partition-PGPROC.patch)
download | inline diff:
From 80415fe4e0de3c642fc15721085ed5af3d6441b1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:10:03 +0100
Subject: [PATCH v20251121 7/7] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 ++-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 551 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 72 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index dc2ce019283..306063e159e 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -33,3 +33,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 179a38fd6ed..cb33bf28b35 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -931,3 +934,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index ea43b432daf..7d589bac115 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -575,7 +575,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -634,7 +634,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 89d0bfa7760..e0e17293536 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..3288900bb6f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -292,7 +292,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..f5844aa5b6a 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 587859a5754..bdcdbcc6b5f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -799,6 +799,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 810a549efce..0937292643f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -472,7 +472,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 9cb78ead105..f82e664ad3f 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..0a0ce98b725 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,22 +29,33 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "access/xlogwait.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -77,8 +88,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -91,6 +102,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -101,11 +135,41 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -130,6 +194,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -141,6 +259,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -149,6 +270,8 @@ ProcGlobalShmemSize(void)
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
+
return size;
}
@@ -193,7 +316,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -212,6 +335,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -226,6 +352,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,21 +376,110 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -291,23 +515,91 @@ InitProcGlobal(void)
/* Reserve space for semaphores. */
PGReserveSemaphores(ProcGlobalSemas());
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
- /* Common initialization for all PGPROCs, regardless of type. */
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -371,9 +663,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -440,7 +729,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -651,7 +984,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1059,7 +1392,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1108,7 +1441,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1998,7 +2331,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2073,3 +2406,173 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ *
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_partition_init procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index fa3bb09effe..c7d0bf10555 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1884,6 +1884,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.1
[text/x-patch] v20251121-0006-NUMA-shared-buffers-partitioning.patch (46.8K, ../../[email protected]/3-v20251121-0006-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From a31e641c96beccea652f4e93ecee22398ff80b15 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:05:35 +0100
Subject: [PATCH v20251121 6/7] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
.../pg_buffercache--1.6--1.7.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 52 +-
src/backend/port/sysv_shmem.c | 32 +
src/backend/storage/buffer/buf_init.c | 569 +++++++++++++++++-
src/backend/storage/buffer/freelist.c | 88 ++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 14 +-
src/include/storage/bufmgr.h | 4 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
12 files changed, 772 insertions(+), 72 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 2c4d560514d..dc2ce019283 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -13,6 +13,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer, -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index aa8ea08e1bb..179a38fd6ed 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -29,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -813,25 +813,27 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -849,7 +851,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -868,7 +871,7 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
@@ -886,36 +889,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
- values[4] = Int64GetDatum(complete_passes);
+ values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
- values[5] = Int32GetDatum(next_victim_buffer);
+ values[5] = Int64GetDatum(complete_passes);
nulls[5] = false;
- values[6] = Int64GetDatum(buffer_total_allocs);
+ values[6] = Int32GetDatum(next_victim_buffer);
nulls[6] = false;
- values[7] = Int64GetDatum(buffer_allocs);
+ values[7] = Int64GetDatum(buffer_total_allocs);
nulls[7] = false;
- values[8] = Int64GetDatum(buffer_total_req_allocs);
+ values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
- values[9] = Int64GetDatum(buffer_req_allocs);
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
nulls[9] = false;
- values[10] = PointerGetDatum(array);
+ values[10] = Int64GetDatum(buffer_req_allocs);
nulls[10] = false;
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..6019bee334d 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include <numa.h>
#include <signal.h>
#include <unistd.h>
#include <sys/file.h>
@@ -602,6 +603,14 @@ CreateAnonymousSegment(Size *size)
void *ptr = MAP_FAILED;
int mmap_errno = 0;
+ /*
+ * Set the memory policy to interleave to all NUMA nodes before calling
+ * mmap, in case we use MAP_POPULATE to prefault all the pages.
+ *
+ * XXX Probably not needed without that, but also costs nothing.
+ */
+ numa_set_membind(numa_all_nodes_ptr);
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -616,6 +625,9 @@ CreateAnonymousSegment(Size *size)
GetHugePageSize(&hugepagesize, &mmap_flags);
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
if (allocsize % hugepagesize != 0)
allocsize += hugepagesize - (allocsize % hugepagesize);
@@ -638,6 +650,11 @@ CreateAnonymousSegment(Size *size)
if (ptr == MAP_FAILED && huge_pages != HUGE_PAGES_ON)
{
+ int mmap_flags = 0;
+
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
/*
* Use the original size, not the rounded-up value, when falling back
* to non-huge pages.
@@ -663,6 +680,21 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+ /* undo the earlier num_set_membind() call. */
+ numa_set_localalloc();
+
+ /*
+ * Before touching the memory, set the allocation policy, so that
+ * it gets interleaved by default. We have to do this to distribute
+ * the memory that's not located explicitly. We need this especially
+ * with huge pages, where we could run out of huge pages on some
+ * nodes and crash otherwise.
+ *
+ * XXX Probably not needed with MAP_POPULATE, in which case the policy
+ * was already set by num_set_membind() earlier. But doesn't hurt.
+ */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 0362fda24aa..587859a5754 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,12 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -29,15 +35,24 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
-/* *
- * number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
/* Array of structs with information about buffer ranges */
BufferPartitions *BufferPartitionsArray = NULL;
+static void buffer_partitions_prepare(void);
static void buffer_partitions_init(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -85,25 +100,85 @@ BufferManagerShmemInit(void)
foundIOCV,
foundBufCkpt,
foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
/* allocate the partition registry first */
BufferPartitionsArray = (BufferPartitions *)
ShmemInitStruct("Buffer Partitions",
offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ mul_size(sizeof(BufferPartition), numa_partitions),
&foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -133,7 +208,10 @@ BufferManagerShmemInit(void)
{
int i;
- /* Initialize buffer partitions (calculate buffer ranges). */
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
buffer_partitions_init();
/*
@@ -172,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -201,11 +286,244 @@ BufferManagerShmemSize(void)
/* account for registry of NUMA partitions */
size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+ mul_size(sizeof(BufferPartition), numa_partitions)));
return size;
}
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches earlier in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
/*
* Sanity checks of buffers partitions - there must be no gaps, it must cover
* the whole range of buffers, etc.
@@ -267,33 +585,137 @@ buffer_partitions_init(void)
{
int remaining_buffers = NBuffers;
int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers
- = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
-
- BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
- for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[n];
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
- /* buffers this partition should get (last partition can get fewer) */
- int num_buffers = Min(remaining_buffers, part_buffers);
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
- remaining_buffers -= num_buffers;
+ remaining_buffers -= node_buffers;
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- buffer += num_buffers;
+ /* XXX we should get the actual node ID from the mask */
+ if (numa_can_partition)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
}
AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ char *startptr,
+ *endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
}
int
@@ -302,14 +724,21 @@ BufferPartitionCount(void)
return BufferPartitionsArray->npartitions;
}
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
{
BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -322,8 +751,82 @@ BufferPartitionGet(int idx, int *num_buffers,
/* return parameters before the partitions are initialized (during sizing) */
void
-BufferPartitionParams(int *num_partitions)
+BufferPartitionParams(int *num_partitions, int *num_nodes)
{
if (num_partitions)
- *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 8be77a9c8b1..810a549efce 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -124,7 +124,9 @@ typedef struct
//int __attribute__((aligned(64))) bgwprocno;
/* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
int num_partitions;
+ int num_partitions_per_node;
/* clocksweep partitions */
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
@@ -270,16 +272,72 @@ ClockSweepTick(ClockSweep *sweep)
* calculate_partition_index
* calculate the buffer / clock-sweep partition to use
*
- * use PID to determine the buffer partition
- *
- * XXX We could use NUMA node / core ID to pick partition, but we'd need
- * to handle cases with fewer nodes/cores than partitions somehow. Although,
- * maybe the balancing would handle that too.
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
*/
static int
calculate_partition_index(void)
{
- return (MyProcPid % StrategyControl->num_partitions);
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
}
/*
@@ -947,7 +1005,7 @@ StrategyShmemSize(void)
Size size = 0;
int num_partitions;
- BufferPartitionParams(&num_partitions);
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -974,9 +1032,17 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
int num_partitions;
+ int num_partitions_per_node;
num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -1011,7 +1077,8 @@ StrategyInitialize(bool init)
/* Initialize the clock sweep pointers (for all partitions) */
for (int i = 0; i < num_partitions; i++)
{
- int num_buffers,
+ int node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -1020,7 +1087,8 @@ StrategyInitialize(bool init)
pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
/* get info about the buffer partition */
- BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
/*
* FIXME This may not quite right, because if NBuffers is not a
@@ -1056,6 +1124,8 @@ StrategyInitialize(bool init)
/* initialize the partitioned clocksweep */
StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 1128167c025..8192c27066b 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -636,6 +636,16 @@
options => 'debug_logical_replication_streaming_options',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'debug_parallel_query', type => 'enum', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Forces the planner\'s use parallel query nodes.',
long_desc => 'This can be useful for testing the parallel query infrastructure by forcing the planner to generate plans that contain nodes that perform tuple communication between workers and the main process.',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0209b2067a2..404eb3432f9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1118b386228..33377841c57 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -299,10 +299,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -312,7 +312,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -555,8 +555,8 @@ extern void AtEOXact_LocalBuffers(bool isCommit);
extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
-extern void BufferPartitionParams(int *num_partitions);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4e7b1fcd4ab..510018db115 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,10 +156,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -169,6 +171,7 @@ typedef struct BufferPartition
typedef struct BufferPartitions
{
int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -346,6 +349,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 540ada3f8ef..d9c3841e078 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -116,6 +119,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -138,4 +171,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
--
2.51.1
[text/x-patch] v20251121-0005-clock-sweep-weighted-balancing.patch (5.2K, ../../[email protected]/4-v20251121-0005-clock-sweep-weighted-balancing.patch)
download | inline diff:
From 1f98649b8db51c26d6488fc3cd30de8298518800 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251121 5/7] clock-sweep: weighted balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
Note: This may be more important with NUMA-aware partitioning, which
restricts the allowed sizes of partiions (especially with huge pages).
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 3af82e267c6..8be77a9c8b1 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -619,6 +619,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -645,16 +659,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -662,8 +687,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -726,6 +757,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -733,7 +768,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -747,22 +782,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.1
[text/x-patch] v20251121-0004-clock-sweep-scan-all-partitions.patch (6.7K, ../../[email protected]/5-v20251121-0004-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From 4cbe8788366287b0347f006ee90bf1470af74877 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251121 4/7] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 169071032b4..3af82e267c6 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -167,6 +167,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -201,10 +204,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -370,7 +372,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -424,37 +427,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -482,7 +517,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -501,7 +536,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.1
[text/x-patch] v20251121-0003-clock-sweep-balancing-of-allocations.patch (25.3K, ../../[email protected]/6-v20251121-0003-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From ee6b06ec3c011eb4530d39f071b2b2c09416cabc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:45:34 +0100
Subject: [PATCH v20251121 3/7] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 14e750beeff..2c4d560514d 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -21,7 +21,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 11c22dda9b3..aa8ea08e1bb 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -794,6 +796,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -823,6 +827,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,11 +853,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -856,8 +872,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -883,6 +907,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffer_allocs);
nulls[7] = false;
+ values[8] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[8] = false;
+
+ values[9] = Int64GetDatum(buffer_req_allocs);
+ nulls[9] = false;
+
+ values[10] = PointerGetDatum(array);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a3092ce801d..82c645a3b00 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3912,6 +3912,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d40b09f7e69..169071032b4 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -130,7 +166,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -142,7 +204,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -233,11 +295,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -309,7 +419,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -417,6 +527,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -443,6 +771,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -627,7 +956,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1001,8 +1344,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1010,11 +1355,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3307190f611..1118b386228 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,6 +508,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7052f9de57c..4e7b1fcd4ab 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -360,11 +360,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.1
[text/x-patch] v20251121-0002-clock-sweep-basic-partitioning.patch (33.4K, ../../[email protected]/7-v20251121-0002-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 920df09b7e585a44c2a528613786527461c5581c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:03:32 +0100
Subject: [PATCH v20251121 2/7] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 8 +
src/backend/storage/buffer/bufmgr.c | 186 ++++++++----
src/backend/storage/buffer/freelist.c | 283 +++++++++++++++---
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 430 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index f1c20960b7e..14e750beeff 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -15,7 +15,13 @@ CREATE VIEW pg_buffercache_partitions AS
(partition integer, -- partition index
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index f11ef46dbed..11c22dda9b3 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -815,6 +815,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -835,12 +843,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -853,6 +871,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[3] = Int32GetDatum(last_buffer);
nulls[3] = false;
+ values[4] = Int64GetDatum(complete_passes);
+ nulls[4] = false;
+
+ values[5] = Int32GetDatum(next_victim_buffer);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffer_total_allocs);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_allocs);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 528a368a8b7..0362fda24aa 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -319,3 +319,11 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions)
+{
+ if (num_partitions)
+ *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 327ddb7adc8..a3092ce801d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3608,33 +3608,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3662,25 +3658,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3692,17 +3679,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3710,11 +3697,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3734,9 +3721,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3750,15 +3737,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3766,9 +3754,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3778,7 +3766,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3786,10 +3774,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3816,7 +3804,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3841,20 +3829,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3868,7 +3856,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3898,8 +3886,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 28d952b3534..d40b09f7e69 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +130,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +142,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +150,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +183,61 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * use PID to determine the buffer partition
+ *
+ * XXX We could use NUMA node / core ID to pick partition, but we'd need
+ * to handle cases with fewer nodes/cores than partitions somehow. Although,
+ * maybe the balancing would handle that too.
+ */
+static int
+calculate_partition_index(void)
+{
+ return (MyProcPid % StrategyControl->num_partitions);
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +309,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +417,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +464,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +538,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +548,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +567,10 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_partitions;
+
+ num_partitions = BufferPartitionCount();
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +588,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +601,40 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
}
else
Assert(!init);
@@ -803,3 +998,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 139055a4a7d..3307190f611 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,7 +508,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -554,5 +556,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 24860c6c2c4..7052f9de57c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -359,6 +359,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4bc3c31d638..fa3bb09effe 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.1
[text/x-patch] v20251121-0001-Infrastructure-for-partitioning-shared-buf.patch (14.9K, ../../[email protected]/8-v20251121-0001-Infrastructure-for-partitioning-shared-buf.patch)
download | inline diff:
From 9aef8aa328dda058ba3ea2e0dc4c65297ea09c3e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251121 1/7] Infrastructure for partitioning shared buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep).
The registry is a small BufferPartitions array in shared memory, with
partitions sized to be a fair share of shared buffers. Later patches may
improve this to consider NUMA, and similar details.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
---
contrib/pg_buffercache/Makefile | 2 +-
contrib/pg_buffercache/meson.build | 1 +
.../pg_buffercache--1.6--1.7.sql | 25 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 144 +++++++++++++++++-
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
9 files changed, 284 insertions(+), 3 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.build
index 7cd039a1df9..7c31141881f 100644
--- a/contrib/pg_buffercache/meson.build
+++ b/contrib/pg_buffercache/meson.build
@@ -24,6 +24,7 @@ install_data(
'pg_buffercache--1.3--1.4.sql',
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
+ 'pg_buffercache--1.6--1.7.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..f1c20960b7e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,25 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index c29b784dfa1..f11ef46dbed 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -776,3 +778,87 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..528a368a8b7 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -17,6 +17,11 @@
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +29,14 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/* *
+ * number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_init(void);
/*
* Data Structures:
@@ -70,7 +83,15 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ &foundParts);
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -112,6 +133,9 @@ BufferManagerShmemInit(void)
{
int i;
+ /* Initialize buffer partitions (calculate buffer ranges). */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -175,5 +199,123 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+
return size;
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers
+ = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[n];
+
+ /* buffers this partition should get (last partition can get fewer) */
+ int num_buffers = Min(remaining_buffers, part_buffers);
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5400c56a965..139055a4a7d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -345,6 +345,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -549,4 +550,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b5f8f3c5d42..24860c6c2c4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -153,6 +153,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 27a4d131897..4bc3c31d638 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-25 14:12 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-11-25 14:12 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
Hi Tomas!
[..]
> Which I think is mostly the same thing you're saying, and you have the maps to support it.
Right, the thread is kind of long, you were right back then, well but
at least we've got a solid explanation with data.
> Here's an updated version of the patch series.
Just for double confirmation, I've used those ones (v20251121*) and
they indeed interleaved parts of shm memory.
> It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
> and a incorrect array length.
You'll need to rebase again, pg_buffercache_numa got updated again on
Monday and clashes with 0006.
> The main change is in 0006 - it sets the default allocation policy for
> shmem to interleaving, before doing the explicit partitioning for shared
> buffers. It does it by calling numa_set_membind before the mmap(), and
> then numa_interleave_memory() on the allocated shmem. It does this to
> allow using MAP_POPULATE - but that's commented out by default.
>
> This does seem to solve the SIGBUS failures for me. I still think there
> might be a small chance of hitting that, because of locating an extra
> "boundary" page on one of the nodes. But it should be solvable by
> reserving a couple more pages.
I can confirm, never got any SIGBUS during the later described
benchmarks, so it's much better now.
> Jakub, what do you think?
On one side not using MAP_POPULATE gives instant startup, but on the
other it gives much better predictability latencies especially fresh
after starting up (this might matter to folks who like to benchmark --
us?, but initially I've just used it as a simple hack to touch
memory). I would be wary of using MAP_POPULATE with s_b when it would
be sized in hundreths of GBs, it could take minutes in startup, which
would be terrible if someone would hit SIGSEGV on production and
expect restart_after_crash=true to save him. I mean WAL redo crash
would be terrible, but that would be terrible * 2. Also pretty
long-term with DIO, we'll get much bigger s_b anyway (hopefully), so
it would hurt even more, so I think that would be a bad path(?)
I've benchmarked the thing in two scenarios (readonly pgbench < s_b
size across variations of code and connections and 2nd one with
seqconcurrrentscans) in solid stable conditions: 4s32c64t == 4 NUMA
nodes, 128GB RAM, 31GB shared_buffers dbsize ~29GB, 6.14.x, no idle
CPU states, no turbo boost, and so on, literally great home heater
when there's -3C outside!)
The data is baseline "100%" for master along with HP on/off (so it's
showing diff % from respective HP setting):
scenario I: pgbench -S
connections
branch HP 1 8 64 128 1024
master off 100.00% 100.00% 100.00% 100.00% 100.00%
master on 100.00% 100.00% 100.00% 100.00% 100.00%
numa16 off 99.13% 100.46% 99.66% 99.44% 89.60%
numa16 on 101.80% 100.89% 99.36% 99.89% 93.43%
numa4 off 96.82% 100.61% 99.37% 99.92% 94.41%
numa4 on 101.83% 100.61% 99.35% 99.69% 101.48%
pgproc16 off 99.13% 100.84% 99.38% 99.85% 91.15%
pgproc16 on 101.72% 101.40% 99.72% 100.14% 95.20%
pgproc4 off 98.63% 101.44% 100.05% 100.14% 90.97%
pgproc4 on 101.05% 101.46% 99.92% 100.31% 97.60%
sweep16 off 99.53% 101.14% 100.71% 100.75% 101.52%
sweep16 on 97.63% 102.49% 100.42% 100.75% 105.56%
sweep4 off 99.43% 101.59% 100.06% 100.45% 104.63%
sweep4 on 97.69% 101.59% 100.70% 100.69% 104.70%
I would consider everything +/- 3% as noise (technically each branch
was a different compilation/ELF binary, as changing this #define
required to do so to get 4 vs 16; please see attached script). I miss
the explanation why without HP it deteriorates so much with for c=1024
with the patches.
scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
without PQ by:
\set num (:client_id % 8) + 1
select sum(octet_length(filler)) from pgbench_accounts_:num;
connections
branch HP 1 8 64 128
master off 100.00% 100.00% 100.00% 100.00%
master on 100.00% 100.00% 100.00% 100.00%
numa16 off 115.62% 108.87% 101.08% 111.56%
numa16 on 107.68% 104.90% 102.98% 105.51%
numa4 off 113.55% 111.41% 101.45% 113.10%
numa4 on 107.90% 106.60% 103.68% 106.98%
pgproc16 off 111.70% 108.27% 98.69% 109.36%
pgproc16 on 106.98% 100.69% 101.98% 103.42%
pgproc4 off 112.41% 106.15% 100.03% 112.03%
pgproc4 on 106.73% 105.77% 103.74% 101.13%
sweep16 off 100.63% 100.38% 98.41% 103.46%
sweep16 on 109.03% 99.15% 101.17% 99.19%
sweep4 off 102.04% 101.16% 101.71% 91.86%
sweep4 on 108.33% 101.69% 97.14% 100.92%
The benefit varies with like +3-10% depending on connection count.
Quite frankly I was expecting a little bit more, especially after
re-reading [1]. Maybe you preloaded it there using pg_prewarm? (here
I've randomly warmed it using pgbench). Probably it's something with
my test, I'll take yet another look hopefully soon. The good thing is
that it never crashed and I haven't seen any errors like "Bad address"
probably related to AIO as you saw in [1], perhaps I wasn't using
uring.
0007 (PROCs) still complains with "mbind: Invalid argument" (aligment issue)
-J.
[1] - https://www.postgresql.org/message-id/e4d7e6fc-b5c5-4288-991c-56219db2edd5%40vondra.me
master 128 off 63.323369
master 128 on 70.424227
master 1 off 1.257394
master 1 on 1.355974
master 64 off 83.681932
master 64 on 84.667119
master 8 off 10.096653
master 8 on 10.801311
numa16 128 off 70.642176
numa16 128 on 74.301325
numa16 1 off 1.453782
numa16 1 on 1.460134
numa16 64 off 84.581887
numa16 64 on 87.187862
numa16 8 off 10.991734
numa16 8 on 11.330359
numa4 128 off 71.619885
numa4 128 on 75.336443
numa4 1 off 1.427726
numa4 1 on 1.463109
numa4 64 off 84.894327
numa4 64 on 87.784820
numa4 8 off 11.248773
numa4 8 on 11.514226
pgproc16 128 off 69.251663
pgproc16 128 on 72.836041
pgproc16 1 off 1.404562
pgproc16 1 on 1.450629
pgproc16 64 off 82.583295
pgproc16 64 on 86.340406
pgproc16 8 off 10.931756
pgproc16 8 on 10.876293
pgproc4 128 off 70.943775
pgproc4 128 on 71.220691
pgproc4 1 off 1.413467
pgproc4 1 on 1.447225
pgproc4 64 off 83.706755
pgproc4 64 on 87.837412
pgproc4 8 off 10.717336
pgproc4 8 on 11.424055
sweep16 128 off 65.516788
sweep16 128 on 69.852675
sweep16 1 off 1.265357
sweep16 1 on 1.478456
sweep16 64 off 82.351384
sweep16 64 on 85.657325
sweep16 8 off 10.134920
sweep16 8 on 10.709992
sweep4 128 off 58.171417
sweep4 128 on 71.074758
sweep4 1 off 1.283089
sweep4 1 on 1.468863
sweep4 64 off 85.115087
sweep4 64 on 82.245140
sweep4 8 off 10.214124
sweep4 8 on 10.984077
numa16 1024 off 156531
numa16 1024 on 203669
numa16 128 off 304947
numa16 128 on 341334
numa16 1 off 5954
numa16 1 on 3785
numa16 64 off 320373
numa16 64 on 350338
numa16 8 off 47256
numa16 8 on 51998
numa4 1024 off 164933
numa4 1024 on 221202
numa4 128 off 306432
numa4 128 on 340640
numa4 1 off 5815
numa4 1 on 3786
numa4 64 off 319466
numa4 64 on 350299
numa4 8 off 47325
numa4 8 on 51851
sweep16 1024 off 177348
sweep16 1024 on 230097
sweep16 128 off 308967
sweep16 128 on 344263
sweep16 1 off 5978
sweep16 1 on 3630
sweep16 64 off 323757
sweep16 64 on 354071
sweep16 8 off 47575
sweep16 8 on 52821
sweep4 1024 off 182788
sweep4 1024 on 228218
sweep4 128 off 308046
sweep4 128 on 344065
sweep4 1 off 5972
sweep4 1 on 3632
sweep4 64 off 321674
sweep4 64 on 355082
sweep4 8 off 47785
sweep4 8 on 52357
master 1024 off 174692
master 1024 on 217981
master 128 off 306671
master 128 on 341712
master 1 off 6006
master 1 on 3718
master 64 off 321482
master 64 on 352603
master 8 off 47039
master 8 on 51537
pgproc16 1024 off 159239
pgproc16 1024 on 207525
pgproc16 128 off 306219
pgproc16 128 on 342183
pgproc16 1 off 5954
pgproc16 1 on 3782
pgproc16 64 off 319485
pgproc16 64 on 351629
pgproc16 8 off 47434
pgproc16 8 on 52259
pgproc4 1024 off 158922
pgproc4 1024 on 212753
pgproc4 128 off 307103
pgproc4 128 on 342786
pgproc4 1 off 5924
pgproc4 1 on 3757
pgproc4 64 off 321649
pgproc4 64 on 352308
pgproc4 8 off 47717
pgproc4 8 on 52292
Attachments:
[text/plain] numa-v20251121-seqconcurrscans.txt (1.4K, ../../CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com/2-numa-v20251121-seqconcurrscans.txt)
download | inline:
master 128 off 63.323369
master 128 on 70.424227
master 1 off 1.257394
master 1 on 1.355974
master 64 off 83.681932
master 64 on 84.667119
master 8 off 10.096653
master 8 on 10.801311
numa16 128 off 70.642176
numa16 128 on 74.301325
numa16 1 off 1.453782
numa16 1 on 1.460134
numa16 64 off 84.581887
numa16 64 on 87.187862
numa16 8 off 10.991734
numa16 8 on 11.330359
numa4 128 off 71.619885
numa4 128 on 75.336443
numa4 1 off 1.427726
numa4 1 on 1.463109
numa4 64 off 84.894327
numa4 64 on 87.784820
numa4 8 off 11.248773
numa4 8 on 11.514226
pgproc16 128 off 69.251663
pgproc16 128 on 72.836041
pgproc16 1 off 1.404562
pgproc16 1 on 1.450629
pgproc16 64 off 82.583295
pgproc16 64 on 86.340406
pgproc16 8 off 10.931756
pgproc16 8 on 10.876293
pgproc4 128 off 70.943775
pgproc4 128 on 71.220691
pgproc4 1 off 1.413467
pgproc4 1 on 1.447225
pgproc4 64 off 83.706755
pgproc4 64 on 87.837412
pgproc4 8 off 10.717336
pgproc4 8 on 11.424055
sweep16 128 off 65.516788
sweep16 128 on 69.852675
sweep16 1 off 1.265357
sweep16 1 on 1.478456
sweep16 64 off 82.351384
sweep16 64 on 85.657325
sweep16 8 off 10.134920
sweep16 8 on 10.709992
sweep4 128 off 58.171417
sweep4 128 on 71.074758
sweep4 1 off 1.283089
sweep4 1 on 1.468863
sweep4 64 off 85.115087
sweep4 64 on 82.245140
sweep4 8 off 10.214124
sweep4 8 on 10.984077
[application/x-shellscript] bench_numa.sh (2.4K, ../../CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com/3-bench_numa.sh)
download
[text/plain] numa-v20251121-pgbenchS.txt (1.5K, ../../CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com/4-numa-v20251121-pgbenchS.txt)
download | inline:
numa16 1024 off 156531
numa16 1024 on 203669
numa16 128 off 304947
numa16 128 on 341334
numa16 1 off 5954
numa16 1 on 3785
numa16 64 off 320373
numa16 64 on 350338
numa16 8 off 47256
numa16 8 on 51998
numa4 1024 off 164933
numa4 1024 on 221202
numa4 128 off 306432
numa4 128 on 340640
numa4 1 off 5815
numa4 1 on 3786
numa4 64 off 319466
numa4 64 on 350299
numa4 8 off 47325
numa4 8 on 51851
sweep16 1024 off 177348
sweep16 1024 on 230097
sweep16 128 off 308967
sweep16 128 on 344263
sweep16 1 off 5978
sweep16 1 on 3630
sweep16 64 off 323757
sweep16 64 on 354071
sweep16 8 off 47575
sweep16 8 on 52821
sweep4 1024 off 182788
sweep4 1024 on 228218
sweep4 128 off 308046
sweep4 128 on 344065
sweep4 1 off 5972
sweep4 1 on 3632
sweep4 64 off 321674
sweep4 64 on 355082
sweep4 8 off 47785
sweep4 8 on 52357
master 1024 off 174692
master 1024 on 217981
master 128 off 306671
master 128 on 341712
master 1 off 6006
master 1 on 3718
master 64 off 321482
master 64 on 352603
master 8 off 47039
master 8 on 51537
pgproc16 1024 off 159239
pgproc16 1024 on 207525
pgproc16 128 off 306219
pgproc16 128 on 342183
pgproc16 1 off 5954
pgproc16 1 on 3782
pgproc16 64 off 319485
pgproc16 64 on 351629
pgproc16 8 off 47434
pgproc16 8 on 52259
pgproc4 1024 off 158922
pgproc4 1024 on 212753
pgproc4 128 off 307103
pgproc4 128 on 342786
pgproc4 1 off 5924
pgproc4 1 on 3757
pgproc4 64 off 321649
pgproc4 64 on 352308
pgproc4 8 off 47717
pgproc4 8 on 52292
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-11-26 16:19 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-11-26 16:19 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On 11/25/25 15:12, Jakub Wartak wrote:
> Hi Tomas!
>
> [..]
>> Which I think is mostly the same thing you're saying, and you have the maps to support it.
>
> Right, the thread is kind of long, you were right back then, well but
> at least we've got a solid explanation with data.
>
>> Here's an updated version of the patch series.
>
> Just for double confirmation, I've used those ones (v20251121*) and
> they indeed interleaved parts of shm memory.
>
>> It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
>> and a incorrect array length.
>
> You'll need to rebase again, pg_buffercache_numa got updated again on
> Monday and clashes with 0006.
>
Rebased patch series attached.
>> The main change is in 0006 - it sets the default allocation policy for
>> shmem to interleaving, before doing the explicit partitioning for shared
>> buffers. It does it by calling numa_set_membind before the mmap(), and
>> then numa_interleave_memory() on the allocated shmem. It does this to
>> allow using MAP_POPULATE - but that's commented out by default.
>>
>> This does seem to solve the SIGBUS failures for me. I still think there
>> might be a small chance of hitting that, because of locating an extra
>> "boundary" page on one of the nodes. But it should be solvable by
>> reserving a couple more pages.
>
> I can confirm, never got any SIGBUS during the later described
> benchmarks, so it's much better now.
>
Good!
>> Jakub, what do you think?
>
> On one side not using MAP_POPULATE gives instant startup, but on the
> other it gives much better predictability latencies especially fresh
> after starting up (this might matter to folks who like to benchmark --
> us?, but initially I've just used it as a simple hack to touch
> memory). I would be wary of using MAP_POPULATE with s_b when it would
> be sized in hundreths of GBs, it could take minutes in startup, which
> would be terrible if someone would hit SIGSEGV on production and
> expect restart_after_crash=true to save him. I mean WAL redo crash
> would be terrible, but that would be terrible * 2. Also pretty
> long-term with DIO, we'll get much bigger s_b anyway (hopefully), so
> it would hurt even more, so I think that would be a bad path(?)
>
I think the MAP_POPULATE should be optional, enabled by GUC.
> I've benchmarked the thing in two scenarios (readonly pgbench < s_b
> size across variations of code and connections and 2nd one with
> seqconcurrrentscans) in solid stable conditions: 4s32c64t == 4 NUMA
> nodes, 128GB RAM, 31GB shared_buffers dbsize ~29GB, 6.14.x, no idle
> CPU states, no turbo boost, and so on, literally great home heater
> when there's -3C outside!)
>
> The data is baseline "100%" for master along with HP on/off (so it's
> showing diff % from respective HP setting):
>
> scenario I: pgbench -S
>
> connections
> branch HP 1 8 64 128 1024
> master off 100.00% 100.00% 100.00% 100.00% 100.00%
> master on 100.00% 100.00% 100.00% 100.00% 100.00%
> numa16 off 99.13% 100.46% 99.66% 99.44% 89.60%
> numa16 on 101.80% 100.89% 99.36% 99.89% 93.43%
> numa4 off 96.82% 100.61% 99.37% 99.92% 94.41%
> numa4 on 101.83% 100.61% 99.35% 99.69% 101.48%
> pgproc16 off 99.13% 100.84% 99.38% 99.85% 91.15%
> pgproc16 on 101.72% 101.40% 99.72% 100.14% 95.20%
> pgproc4 off 98.63% 101.44% 100.05% 100.14% 90.97%
> pgproc4 on 101.05% 101.46% 99.92% 100.31% 97.60%
> sweep16 off 99.53% 101.14% 100.71% 100.75% 101.52%
> sweep16 on 97.63% 102.49% 100.42% 100.75% 105.56%
> sweep4 off 99.43% 101.59% 100.06% 100.45% 104.63%
> sweep4 on 97.69% 101.59% 100.70% 100.69% 104.70%
>
> I would consider everything +/- 3% as noise (technically each branch
> was a different compilation/ELF binary, as changing this #define
> required to do so to get 4 vs 16; please see attached script). I miss
> the explanation why without HP it deteriorates so much with for c=1024
> with the patches.
I wouldn't expect a big difference for "pgbench -S". That workload has
so much other fairly expensive stuff (e.g. initializing index scans
etc.), the cost of buffer replacement is going to be fairly limited.
The regressions for numa/pgproc patches with 1024 clients are annoying,
but how realistic is such scenario? With 32/64 CPUs, having 1024 active
connections is a substantial overload. If we can fix this, great. But I
think such regression may be OK if we get benefits for reasonable setups
(with fewer clients).
I don't know why it's happening, though. I haven't been testing cases
with so many clients (compared to number of CPUs).
>
> scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
> pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
> without PQ by:
> \set num (:client_id % 8) + 1
> select sum(octet_length(filler)) from pgbench_accounts_:num;
>
> connections
> branch HP 1 8 64 128
> master off 100.00% 100.00% 100.00% 100.00%
> master on 100.00% 100.00% 100.00% 100.00%
> numa16 off 115.62% 108.87% 101.08% 111.56%
> numa16 on 107.68% 104.90% 102.98% 105.51%
> numa4 off 113.55% 111.41% 101.45% 113.10%
> numa4 on 107.90% 106.60% 103.68% 106.98%
> pgproc16 off 111.70% 108.27% 98.69% 109.36%
> pgproc16 on 106.98% 100.69% 101.98% 103.42%
> pgproc4 off 112.41% 106.15% 100.03% 112.03%
> pgproc4 on 106.73% 105.77% 103.74% 101.13%
> sweep16 off 100.63% 100.38% 98.41% 103.46%
> sweep16 on 109.03% 99.15% 101.17% 99.19%
> sweep4 off 102.04% 101.16% 101.71% 91.86%
> sweep4 on 108.33% 101.69% 97.14% 100.92%
>
> The benefit varies with like +3-10% depending on connection count.
> Quite frankly I was expecting a little bit more, especially after
> re-reading [1]. Maybe you preloaded it there using pg_prewarm? (here
> I've randomly warmed it using pgbench). Probably it's something with
> my test, I'll take yet another look hopefully soon. The good thing is
> that it never crashed and I haven't seen any errors like "Bad address"
> probably related to AIO as you saw in [1], perhaps I wasn't using
> uring.
>
Hmmm. I'd have expected better results for this workload. So I tried
re-running my seqscan benchmark on the 176-core instance, and I got this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 43 35 40 53 53 46 45
96 55 54 42 47 57 58 53 53
128 59 59 46 50 58 58 57 60
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 98% 79% 92% 122% 122% 105% 104%
96 99% 76% 86% 104% 105% 97% 97%
128 99% 77% 84% 98% 98% 97% 101%
I did the benchmark for individual parts of the patch series. There's a
clear (~20%) speedup for 0005, but 0006 and 0007 make it go away. The
0002/0003 regress it quite a bit. And with 128 clients there's no
improvement at all.
This was with the default number of partitions (i.e. 4). If I increase
the number to 16, I get this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 43 69 82 87 87 78 79
96 55 54 65 85 91 91 86 86
128 59 59 66 77 83 83 82 86
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 99% 158% 189% 199% 199% 180% 180%
96 100% 119% 156% 167% 167% 157% 158%
128 99% 112% 130% 140% 140% 139% 145%
And with 32 partitions, I get this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 44 88 91 90 90 84 84
96 55 54 89 93 93 92 90 91
128 59 59 85 84 86 85 88 87
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 100% 202% 208% 207% 207% 193% 193%
96 100% 163% 169% 171% 168% 165% 166%
128 99% 144% 142% 146% 144% 149% 146%
Those are clearly much better results, so I guess the default number of
partitions may be too low.
What bothers me is that this seems like a very narrow benchmark. I mean,
few systems are doing concurrent seqscans putting this much pressure on
buffer replacement. And once the plans start to do other stuff, the
contention on clock sweep seems to go down substantially (as shown by
the read-only pgbench). So the question is - is this really worth it?
> 0007 (PROCs) still complains with "mbind: Invalid argument" (aligment issue)
>
Should be fixed by the attached patches. The 0006 patch has an issue
with mbind too, but it was visible only when the buffers were not a nice
multiple of memory pages (and multiples of 1GB are fine).
This also moves the memset() until after placing the PGPROC partitions
to different NUMA nodes.
The results above are from v20251121. I'll rerun the tests with the nw
version of the patches. But it can only change the 0006/0007 results, of
course. The 0001-0005 are the same.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20251126-0009-mbind-procs.patch (2.4K, ../../[email protected]/2-v20251126-0009-mbind-procs.patch)
download | inline diff:
From 6a7b5f0c48b861a78baafad699802752edfce669 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 26 Nov 2025 17:13:09 +0100
Subject: [PATCH v20251126 9/9] mbind: procs
---
src/backend/storage/lmgr/proc.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 0a0ce98b725..3d76a3a6429 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -206,11 +206,14 @@ FastPathLockShmemSize(void)
* when not strictly needed (if it's already aligned). And we always
* assume we'll add a whole page, even if the alignment needs only less
* memory.
+ *
+ * XXX We need two extra pages. One for the non-NUMA part (aux processes),
+ * and one to keep the size of the last chunk aligned too.
*/
if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
Assert(numa_nodes > 0);
- size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ size = add_size(size, mul_size((numa_nodes + 2), numa_page_size));
}
return size;
@@ -324,6 +327,7 @@ InitProcGlobal(void)
/* Used for setup of per-backend fast-path slots. */
char *fpPtr,
+ *fpPtrOrig,
*fpEndPtr PG_USED_FOR_ASSERTS_ONLY;
Size fpLockBitsSize,
fpRelIdSize;
@@ -507,7 +511,7 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(fpPtr, 0, requestSize);
+ fpPtrOrig = fpPtr;
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
@@ -595,6 +599,9 @@ InitProcGlobal(void)
Assert(fpPtr <= fpEndPtr);
}
+ /* zero the memory only after locating the memory to NUMA nodes */
+ MemSet(fpPtrOrig, 0, requestSize);
+
for (i = 0; i < TotalProcs; i++)
{
PGPROC *proc = procs[i];
@@ -2521,7 +2528,10 @@ fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
* memory, to make sure it's not mapped to any node yet
*/
if (node != -1)
+ {
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
pg_numa_move_to_node(ptr, endptr, node);
+ }
/*
* Now point the PGPROC entries to the fast-path arrays, and also advance
@@ -2550,7 +2560,8 @@ fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
allprocs_index++;
}
- Assert(ptr == endptr);
+ Assert(ptr <= endptr);
+ Assert((node == -1) || (char *) TYPEALIGN(numa_page_size, ptr) == endptr);
return endptr;
}
--
2.51.1
[text/x-patch] v20251126-0008-NUMA-partition-PGPROC.patch (49.2K, ../../[email protected]/3-v20251126-0008-NUMA-partition-PGPROC.patch)
download | inline diff:
From b6e4879a397210aff0f0f720708471568fe99ea4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:10:03 +0100
Subject: [PATCH v20251126 8/9] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.7--1.8.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 ++-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 551 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 72 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 43d2e84f9d2..265c35c8252 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -31,3 +31,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index eae75375152..c6629f767d1 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -105,6 +107,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -981,3 +984,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index ea43b432daf..7d589bac115 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -575,7 +575,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -634,7 +634,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 89d0bfa7760..e0e17293536 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..3288900bb6f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -292,7 +292,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..f5844aa5b6a 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 9ba455488a0..55016cce93f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -824,6 +824,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 810a549efce..0937292643f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -472,7 +472,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 9cb78ead105..f82e664ad3f 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..0a0ce98b725 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,22 +29,33 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "access/xlogwait.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -77,8 +88,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -91,6 +102,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -101,11 +135,41 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -130,6 +194,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -141,6 +259,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -149,6 +270,8 @@ ProcGlobalShmemSize(void)
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
+
return size;
}
@@ -193,7 +316,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -212,6 +335,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -226,6 +352,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,21 +376,110 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -291,23 +515,91 @@ InitProcGlobal(void)
/* Reserve space for semaphores. */
PGReserveSemaphores(ProcGlobalSemas());
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
- /* Common initialization for all PGPROCs, regardless of type. */
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -371,9 +663,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -440,7 +729,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -651,7 +984,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1059,7 +1392,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1108,7 +1441,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1998,7 +2331,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2073,3 +2406,173 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ *
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_partition_init procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241f175e9da..e1bf02a3567 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1888,6 +1888,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.1
[text/x-patch] v20251126-0007-mbind-buffers.patch (2.8K, ../../[email protected]/4-v20251126-0007-mbind-buffers.patch)
download | inline diff:
From 191b5b6503c0884260d14479fe5b19fc1cc7cc86 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 26 Nov 2025 17:12:57 +0100
Subject: [PATCH v20251126 7/9] mbind: buffers
---
src/backend/storage/buffer/buf_init.c | 29 +++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 587859a5754..9ba455488a0 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -272,6 +272,13 @@ BufferManagerShmemSize(void)
size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
+ /*
+ * Extra alignment, so that the partitions are whole memory pages (we
+ * may need to pad the last one, so one page is enough). Without this
+ * we may get mbind() failures in pg_numa_move_to_node().
+ */
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
+
/* size of stuff controlled by freelist.c */
size = add_size(size, StrategyShmemSize());
@@ -695,6 +702,15 @@ buffer_partitions_init(void)
/* first map buffers */
startptr = buffers_ptr;
endptr = startptr + ((Size) num_buffers * BLCKSZ);
+
+ /*
+ * Make sure the partition is a multiple of memory page, so that we
+ * don't get mbind failures in move_to_node calls. This matters only
+ * for the last partition, the earlier ones should be always sized
+ * as multiples of pages.
+ */
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+
buffers_ptr = endptr; /* start of the next partition */
elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
@@ -705,6 +721,15 @@ buffer_partitions_init(void)
/* now do the same for buffer descriptors */
startptr = descriptors_ptr;
endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+
+ /*
+ * Make sure the partition is a multiple of memory page, so that we
+ * don't get mbind failures in move_to_node calls. This matters only
+ * for the last partition, the earlier ones should be always sized
+ * as multiples of pages.
+ */
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+
descriptors_ptr = endptr;
elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
@@ -714,8 +739,8 @@ buffer_partitions_init(void)
}
/* we should have consumed the arrays exactly */
- Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
- Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+ Assert(buffers_ptr <= (char *) TYPEALIGN(numa_page_size, BufferBlocks + (Size) NBuffers * BLCKSZ));
+ Assert(descriptors_ptr == (char *) TYPEALIGN(numa_page_size, (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded)));
}
int
--
2.51.1
[text/x-patch] v20251126-0006-NUMA-shared-buffers-partitioning.patch (47.0K, ../../[email protected]/5-v20251126-0006-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 4b3779f92217900c0eea91ef68810867362a7170 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:05:35 +0100
Subject: [PATCH v20251126 6/9] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 52 +-
src/backend/port/sysv_shmem.c | 34 +-
src/backend/storage/buffer/buf_init.c | 569 +++++++++++++++++-
src/backend/storage/buffer/freelist.c | 88 ++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 14 +-
src/include/storage/bufmgr.h | 4 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
12 files changed, 773 insertions(+), 73 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 1834599c4b3..43d2e84f9d2 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer, -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 81665209084..eae75375152 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -29,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -863,25 +863,27 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -899,7 +901,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -918,7 +921,7 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
@@ -936,36 +939,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
- values[4] = Int64GetDatum(complete_passes);
+ values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
- values[5] = Int32GetDatum(next_victim_buffer);
+ values[5] = Int64GetDatum(complete_passes);
nulls[5] = false;
- values[6] = Int64GetDatum(buffer_total_allocs);
+ values[6] = Int32GetDatum(next_victim_buffer);
nulls[6] = false;
- values[7] = Int64GetDatum(buffer_allocs);
+ values[7] = Int64GetDatum(buffer_total_allocs);
nulls[7] = false;
- values[8] = Int64GetDatum(buffer_total_req_allocs);
+ values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
- values[9] = Int64GetDatum(buffer_req_allocs);
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
nulls[9] = false;
- values[10] = PointerGetDatum(array);
+ values[10] = Int64GetDatum(buffer_req_allocs);
nulls[10] = false;
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..78a0c5199f1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include <numa.h>
#include <signal.h>
#include <unistd.h>
#include <sys/file.h>
@@ -602,6 +603,14 @@ CreateAnonymousSegment(Size *size)
void *ptr = MAP_FAILED;
int mmap_errno = 0;
+ /*
+ * Set the memory policy to interleave to all NUMA nodes before calling
+ * mmap, in case we use MAP_POPULATE to prefault all the pages.
+ *
+ * XXX Probably not needed without that, but also costs nothing.
+ */
+ numa_set_membind(numa_all_nodes_ptr);
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -616,6 +625,9 @@ CreateAnonymousSegment(Size *size)
GetHugePageSize(&hugepagesize, &mmap_flags);
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
if (allocsize % hugepagesize != 0)
allocsize += hugepagesize - (allocsize % hugepagesize);
@@ -638,13 +650,18 @@ CreateAnonymousSegment(Size *size)
if (ptr == MAP_FAILED && huge_pages != HUGE_PAGES_ON)
{
+ int mmap_flags = 0;
+
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
/*
* Use the original size, not the rounded-up value, when falling back
* to non-huge pages.
*/
allocsize = *size;
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS, -1, 0);
+ PG_MMAP_FLAGS | mmap_flags, -1, 0);
mmap_errno = errno;
}
@@ -663,6 +680,21 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+ /* undo the earlier num_set_membind() call. */
+ numa_set_localalloc();
+
+ /*
+ * Before touching the memory, set the allocation policy, so that
+ * it gets interleaved by default. We have to do this to distribute
+ * the memory that's not located explicitly. We need this especially
+ * with huge pages, where we could run out of huge pages on some
+ * nodes and crash otherwise.
+ *
+ * XXX Probably not needed with MAP_POPULATE, in which case the policy
+ * was already set by num_set_membind() earlier. But doesn't hurt.
+ */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 0362fda24aa..587859a5754 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,12 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -29,15 +35,24 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
-/* *
- * number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
/* Array of structs with information about buffer ranges */
BufferPartitions *BufferPartitionsArray = NULL;
+static void buffer_partitions_prepare(void);
static void buffer_partitions_init(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -85,25 +100,85 @@ BufferManagerShmemInit(void)
foundIOCV,
foundBufCkpt,
foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
/* allocate the partition registry first */
BufferPartitionsArray = (BufferPartitions *)
ShmemInitStruct("Buffer Partitions",
offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ mul_size(sizeof(BufferPartition), numa_partitions),
&foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -133,7 +208,10 @@ BufferManagerShmemInit(void)
{
int i;
- /* Initialize buffer partitions (calculate buffer ranges). */
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
buffer_partitions_init();
/*
@@ -172,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -201,11 +286,244 @@ BufferManagerShmemSize(void)
/* account for registry of NUMA partitions */
size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+ mul_size(sizeof(BufferPartition), numa_partitions)));
return size;
}
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches earlier in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
/*
* Sanity checks of buffers partitions - there must be no gaps, it must cover
* the whole range of buffers, etc.
@@ -267,33 +585,137 @@ buffer_partitions_init(void)
{
int remaining_buffers = NBuffers;
int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers
- = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
-
- BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
- for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[n];
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
- /* buffers this partition should get (last partition can get fewer) */
- int num_buffers = Min(remaining_buffers, part_buffers);
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
- remaining_buffers -= num_buffers;
+ remaining_buffers -= node_buffers;
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- buffer += num_buffers;
+ /* XXX we should get the actual node ID from the mask */
+ if (numa_can_partition)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
}
AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ char *startptr,
+ *endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
}
int
@@ -302,14 +724,21 @@ BufferPartitionCount(void)
return BufferPartitionsArray->npartitions;
}
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
{
BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -322,8 +751,82 @@ BufferPartitionGet(int idx, int *num_buffers,
/* return parameters before the partitions are initialized (during sizing) */
void
-BufferPartitionParams(int *num_partitions)
+BufferPartitionParams(int *num_partitions, int *num_nodes)
{
if (num_partitions)
- *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 8be77a9c8b1..810a549efce 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -124,7 +124,9 @@ typedef struct
//int __attribute__((aligned(64))) bgwprocno;
/* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
int num_partitions;
+ int num_partitions_per_node;
/* clocksweep partitions */
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
@@ -270,16 +272,72 @@ ClockSweepTick(ClockSweep *sweep)
* calculate_partition_index
* calculate the buffer / clock-sweep partition to use
*
- * use PID to determine the buffer partition
- *
- * XXX We could use NUMA node / core ID to pick partition, but we'd need
- * to handle cases with fewer nodes/cores than partitions somehow. Although,
- * maybe the balancing would handle that too.
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
*/
static int
calculate_partition_index(void)
{
- return (MyProcPid % StrategyControl->num_partitions);
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
}
/*
@@ -947,7 +1005,7 @@ StrategyShmemSize(void)
Size size = 0;
int num_partitions;
- BufferPartitionParams(&num_partitions);
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -974,9 +1032,17 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
int num_partitions;
+ int num_partitions_per_node;
num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -1011,7 +1077,8 @@ StrategyInitialize(bool init)
/* Initialize the clock sweep pointers (for all partitions) */
for (int i = 0; i < num_partitions; i++)
{
- int num_buffers,
+ int node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -1020,7 +1087,8 @@ StrategyInitialize(bool init)
pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
/* get info about the buffer partition */
- BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
/*
* FIXME This may not quite right, because if NBuffers is not a
@@ -1056,6 +1124,8 @@ StrategyInitialize(bool init)
/* initialize the partitioned clocksweep */
StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 3b9d8349078..b13e667ac9e 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -643,6 +643,16 @@
options => 'debug_logical_replication_streaming_options',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'debug_parallel_query', type => 'enum', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Forces the planner\'s use parallel query nodes.',
long_desc => 'This can be useful for testing the parallel query infrastructure by forcing the planner to generate plans that contain nodes that perform tuple communication between workers and the main process.',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f87b558c2c6..3f29eeaf5ae 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1118b386228..33377841c57 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -299,10 +299,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -312,7 +312,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -555,8 +555,8 @@ extern void AtEOXact_LocalBuffers(bool isCommit);
extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
-extern void BufferPartitionParams(int *num_partitions);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4e7b1fcd4ab..510018db115 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,10 +156,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -169,6 +171,7 @@ typedef struct BufferPartition
typedef struct BufferPartitions
{
int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -346,6 +349,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 540ada3f8ef..d9c3841e078 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -116,6 +119,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -138,4 +171,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
--
2.51.1
[text/x-patch] v20251126-0005-clock-sweep-weighted-balancing.patch (5.2K, ../../[email protected]/6-v20251126-0005-clock-sweep-weighted-balancing.patch)
download | inline diff:
From 16013b08d9a524493c455278b1f4d97f237ec99e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251126 5/9] clock-sweep: weighted balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
Note: This may be more important with NUMA-aware partitioning, which
restricts the allowed sizes of partiions (especially with huge pages).
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 3af82e267c6..8be77a9c8b1 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -619,6 +619,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -645,16 +659,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -662,8 +687,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -726,6 +757,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -733,7 +768,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -747,22 +782,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.1
[text/x-patch] v20251126-0004-clock-sweep-scan-all-partitions.patch (6.7K, ../../[email protected]/7-v20251126-0004-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From 5df379b3a4f9773de589120c3a609d2008663620 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251126 4/9] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 169071032b4..3af82e267c6 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -167,6 +167,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -201,10 +204,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -370,7 +372,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -424,37 +427,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -482,7 +517,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -501,7 +536,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.1
[text/x-patch] v20251126-0003-clock-sweep-balancing-of-allocations.patch (25.3K, ../../[email protected]/8-v20251126-0003-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From d5e333e0300db911e499e9bf2a7db95ff46e674d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:45:34 +0100
Subject: [PATCH v20251126 3/9] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index f702a9db1a8..1834599c4b3 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -19,7 +19,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index aed0ecb59e9..81665209084 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -844,6 +846,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -873,6 +877,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -893,11 +903,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -906,8 +922,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -933,6 +957,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffer_allocs);
nulls[7] = false;
+ values[8] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[8] = false;
+
+ values[9] = Int64GetDatum(buffer_req_allocs);
+ nulls[9] = false;
+
+ values[10] = PointerGetDatum(array);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a3092ce801d..82c645a3b00 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3912,6 +3912,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d40b09f7e69..169071032b4 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -130,7 +166,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -142,7 +204,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -233,11 +295,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -309,7 +419,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -417,6 +527,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -443,6 +771,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -627,7 +956,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1001,8 +1344,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1010,11 +1355,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3307190f611..1118b386228 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,6 +508,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7052f9de57c..4e7b1fcd4ab 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -360,11 +360,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.1
[text/x-patch] v20251126-0002-clock-sweep-basic-partitioning.patch (33.4K, ../../[email protected]/9-v20251126-0002-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 670ef95cfbec187fbc3e2cce641dec9bd79a092e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:03:32 +0100
Subject: [PATCH v20251126 2/9] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 8 +
src/backend/storage/buffer/bufmgr.c | 186 ++++++++----
src/backend/storage/buffer/freelist.c | 283 +++++++++++++++---
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 430 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index d62b8339bfc..f702a9db1a8 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -13,7 +13,13 @@ CREATE VIEW pg_buffercache_partitions AS
(partition integer, -- partition index
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8c89855192f..aed0ecb59e9 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -865,6 +865,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -885,12 +893,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -903,6 +921,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[3] = Int32GetDatum(last_buffer);
nulls[3] = false;
+ values[4] = Int64GetDatum(complete_passes);
+ nulls[4] = false;
+
+ values[5] = Int32GetDatum(next_victim_buffer);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffer_total_allocs);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_allocs);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 528a368a8b7..0362fda24aa 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -319,3 +319,11 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions)
+{
+ if (num_partitions)
+ *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 327ddb7adc8..a3092ce801d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3608,33 +3608,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3662,25 +3658,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3692,17 +3679,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3710,11 +3697,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3734,9 +3721,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3750,15 +3737,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3766,9 +3754,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3778,7 +3766,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3786,10 +3774,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3816,7 +3804,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3841,20 +3829,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3868,7 +3856,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3898,8 +3886,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 28d952b3534..d40b09f7e69 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +130,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +142,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +150,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +183,61 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * use PID to determine the buffer partition
+ *
+ * XXX We could use NUMA node / core ID to pick partition, but we'd need
+ * to handle cases with fewer nodes/cores than partitions somehow. Although,
+ * maybe the balancing would handle that too.
+ */
+static int
+calculate_partition_index(void)
+{
+ return (MyProcPid % StrategyControl->num_partitions);
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +309,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +417,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +464,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +538,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +548,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +567,10 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_partitions;
+
+ num_partitions = BufferPartitionCount();
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +588,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +601,40 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
}
else
Assert(!init);
@@ -803,3 +998,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 139055a4a7d..3307190f611 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,7 +508,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -554,5 +556,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 24860c6c2c4..7052f9de57c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -359,6 +359,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d3edff346a8..241f175e9da 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.1
[text/x-patch] v20251126-0001-Infrastructure-for-partitioning-shared-buf.patch (13.6K, ../../[email protected]/10-v20251126-0001-Infrastructure-for-partitioning-shared-buf.patch)
download | inline diff:
From abe7cf9f1314bdaae2361ad237ef859e00b69c07 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251126 1/9] Infrastructure for partitioning shared buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep).
The registry is a small BufferPartitions array in shared memory, with
partitions sized to be a fair share of shared buffers. Later patches may
improve this to consider NUMA, and similar details.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
---
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 144 +++++++++++++++++-
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
7 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
new file mode 100644
index 00000000000..d62b8339bfc
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -0,0 +1,23 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index 11499550945..d2fa8ba53ba 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'
+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae1712fc93c..8c89855192f 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -101,6 +102,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -826,3 +828,87 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..528a368a8b7 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -17,6 +17,11 @@
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +29,14 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/* *
+ * number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_init(void);
/*
* Data Structures:
@@ -70,7 +83,15 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ &foundParts);
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -112,6 +133,9 @@ BufferManagerShmemInit(void)
{
int i;
+ /* Initialize buffer partitions (calculate buffer ranges). */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -175,5 +199,123 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+
return size;
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers
+ = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[n];
+
+ /* buffers this partition should get (last partition can get fewer) */
+ int num_buffers = Min(remaining_buffers, part_buffers);
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5400c56a965..139055a4a7d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -345,6 +345,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -549,4 +550,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b5f8f3c5d42..24860c6c2c4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -153,6 +153,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dfcd619bfee..d3edff346a8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-12-02 12:26 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2025-12-02 12:26 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
On Wed, Nov 26, 2025 at 5:19 PM Tomas Vondra <[email protected]> wrote:
> Rebased patch series attached.
Thanks. BTW still with the old patchset series, One additional thing
that I've found out related to interleave is that in
CreateAnonymousSegment() with the default check_debug='', we still
issue numa_interleave_memory(ptr..). It should be optional (this also
affects earlier calls too). Tiny patch attached.
> I think the MAP_POPULATE should be optional, enabled by GUC.
OK, but you mean it's a new option to debug_numa, right? (not some
separate) so debug_numa='prefault' then?
> > I would consider everything +/- 3% as noise (technically each branch
> > was a different compilation/ELF binary, as changing this #define
> > required to do so to get 4 vs 16; please see attached script). I miss
> > the explanation why without HP it deteriorates so much with for c=1024
> > with the patches.
>
> I wouldn't expect a big difference for "pgbench -S". That workload has
> so much other fairly expensive stuff (e.g. initializing index scans
> etc.), the cost of buffer replacement is going to be fairly limited.
Right. OK, so I've got the seqconcurrentscans comparison done right,
that is when prewarmed and not naturally filled:
@master, 29GB/s mem bandwidth
latency average = 1255.572 ms
latency stddev = 417.162 ms
tps = 50.451925 (without initial connection time)
@v20251121 patchset, 41GB/s (~10GB/s per socket)
latency average = 719.931 ms
latency stddev = 14.874 ms
tps = 88.362091 (without initial connection time)
The main PMC difference seems to be much lower "backend cycles idle"
(51% master and vs 31% for the NUMA debug_numa="buffers,procs", so
less is waiting on memory, thus it gets that speedup and better IPC).
Anyway, the biggest gripe right now (at least to me) is reliable
benchmarking. Below runs are all apples and oranges comparisons (they
measure different stuff although looks the same initially)
- restart and just select pg_shmem_allocations_numa or prewarm puts
everything into 1 NUMA node with check_numa='', because of prefaulting
happening during select-view case
- restart and pgbench -i -s XX (same issue as above) then pgbench -
you get the same, everything on potential one NUMA node (because
pgbench prefaults just on one)
- restart and pgbench -c 64.. with debug_numa='' (off) MIGHT get
random NUMA layout, how's that is supposed to be deterministic? at
least with debug_numa='buffers' you get determinism..
- the shared_buffers size vs size of dataset read, the moment you
start doing something CPU intensive (or like calling syscalls just for
VFS cache), the benefit seems to disappear at least on my hardware
Anyway, depending on the scenario I could get varied results like
34tps .. 88tps here. The debug_numa='buffers,..' gives just assurance
of the proper layout of shared memory is there (one could even argue
that such performance deviations across runs are bug ;)).
> The regressions for numa/pgproc patches with 1024 clients are annoying,
> but how realistic is such scenario? With 32/64 CPUs, having 1024 active
> connections is a substantial overload. If we can fix this, great. But I
> think such regression may be OK if we get benefits for reasonable setups
> (with fewer clients).
>
> I don't know why it's happening, though. I haven't been testing cases
> with so many clients (compared to the number of CPUs).
The only thing in my mind about deterioration of high-connection count
(AKA -c 1024 scenario) with pgprocs, would be related to the question
you raised in 0007 "Note: The scheduler may migrate the process to a
different CPU/node later. Maybe we should consider pinning the process
to the node?"
I think the answer is yes, so to fetch MyProc based on sched_getcpu()
and then maybe with additional numa_flags & new PROCS_PIN_NODE simply
numa_run_on_node(node)? I've tried this:
pgbench -c 1024 -j 64 -P 1 -T 30 -S -M prepared got:
@numa-empty-debug_numa ~434k TPS, ~12k CPU migrations/second
@numa+buffers+pgproc ~412k TPS, 7-8k CPU migrations/second
@numa+buffers+pgproc+pinnode ~434k TPS, still with 7-8k CPU
migrations/second (so same)
but I've verified for the last one, with bpftrace on that
tracepoint:sched:sched_migrate_task did not performed node-to-node
process bounces anymore (it did for pgbench but not for postgres
itself with this numa_run_on_node())
> > scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
> > pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
[..]
>
> Hmmm. I'd have expected better results for this workload. So I tried
> re-running my seqscan benchmark on the 176-core instance, and I got this:
[..]
Thanks!
> I did the benchmark for individual parts of the patch series. There's a
> clear (~20%) speedup for 0005, but 0006 and 0007 make it go away. The
> 0002/0003 regress it quite a bit. And with 128 clients there's no
> improvement at all.
[..]
> Those are clearly much better results, so I guess the default number of
> partitions may be too low.
>
> What bothers me is that this seems like a very narrow benchmark. I mean,
> few systems are doing concurrent seqscans putting this much pressure on
> buffer replacement. And once the plans start to do other stuff, the
> contention on clock sweep seems to go down substantially (as shown by
> the read-only pgbench). So the question is - is this really worth it?
Are you thinking here about whole NUMA patchset or just clocksweep? I
think multiple clocksweep are just not shining because other
bottlenecks hammer the efficiency here. Andres talk about it exactly
here https://youtu.be/V75KpACdl6E?t=1990 (He mentions out of order
execution, I see btrees in reports as top#1). So maybe it's just too
early to see the results of this optimization?
As for classic readonly pgbench -S I still see roughly 1:8 local to
remote (!) DRAM access (1 <-> 3 sockets) even with those patches, so
potentially something could be improved in far future for sure (that
would require some memaddr monitoring for most remote DRAM misses <->
pg inter-shm ptr mapping; think of pg_shmem_allocations_numa with
local/remote counters or maybe just fallback to perf-c2c).
To sum up, IMHO I understand this $thread's NUMA implementation as:
- it's strictly a guard mechanism to get determinism (for most cases)
-- it fixes "imbalance"
- no performance boost for OLTP as such
- for analytics it could be win (in-memory workloads; well PG is not
fully built for this, but it could be one day/or already is with 3rd
party TAMs and extensions), and:
-- we can provide performance jump for seqconcurrentjobs or memory
fitting workloads (patchset does this already). Note: I think PG will
eventually get into such classes in the longer run, we are just ahead
with NUMA, but PG is without proper vectorized executor stuff.
-- we could further enhance PQ here: the leader and PQ workers would
stick to the same NUMA node with some affinity (the earlier thread
measurements for this [1] -- we could have session GUC to enable this
for planned big PQ whole-NUMA SELECTs; this would be probably done
close to dsm_impl_posix())
- new idea: we could allow exposing tables(spaces) into NUMA nodes or
make it per-user toggle too while we are at it (imagine HTAP-like
workloads: NUMA node #0 for OLTP, node #1 for analytics). Sounds cool
and rather easy and has valid use, but dunno if that would be really
useful?
Way out of scope:
- superlocking btress that Andres mentioned on his presentation
-J.
[1] - https://www.postgresql.org/message-id/attachment/178120/NUMA_pq_cpu_pinning_results.txt
Attachments:
[text/x-patch] CreateAnonymousSegment.diff (1.4K, ../../CAKZiRmxwN+qMpbijCLPix_y6mwSjgus2CPPj=1+uFo9fQG-Knw@mail.gmail.com/2-CreateAnonymousSegment.diff)
download | inline diff:
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 6019bee334d..e5c4752d9f6 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -30,6 +30,7 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
+#include "port/pg_numa.h"
#include "portability/mem.h"
#include "storage/dsm.h"
#include "storage/fd.h"
@@ -609,7 +610,8 @@ CreateAnonymousSegment(Size *size)
*
* XXX Probably not needed without that, but also costs nothing.
*/
- numa_set_membind(numa_all_nodes_ptr);
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ numa_set_membind(numa_all_nodes_ptr);
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
@@ -681,7 +683,8 @@ CreateAnonymousSegment(Size *size)
}
/* undo the earlier num_set_membind() call. */
- numa_set_localalloc();
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ numa_set_localalloc();
/*
* Before touching the memory, set the allocation policy, so that
@@ -693,7 +696,8 @@ CreateAnonymousSegment(Size *size)
* XXX Probably not needed with MAP_POPULATE, in which case the policy
* was already set by num_set_membind() earlier. But doesn't hurt.
*/
- numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+ if ((numa_flags & NUMA_BUFFERS) != 0)
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
*size = allocsize;
return ptr;
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2025-12-08 20:02 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2025-12-08 20:02 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>; Andres Freund <[email protected]>
Hi,
I've spent the last couple days considering what to do about this patch
series in this thread. The approach assumes partitioning shared memory
in a NUMA-aware way has enough benefits to justify the extra complexity.
I'm getting somewhat skeptical about that being a good trade off.
We've been able to demonstrate some significant benefits of the patches.
See for example the results from about a month ago [1], showing that in
some cases the throughput almost doubles.
But I'm skeptical about this for two reasons:
* Most of the benefit comes from patches unrelated to NUMA. The initial
patches partition clockweep, in a NUMA oblivious way. In fact, applying
the NUMA patches often *reduces* the throughput. So if we're concerned
about contention on the clocksweep hand, we could apply just these first
patches. That way we wouldn't have to deal with huge pages.
* Furthermore, I'm not quite sure clocksweep really is a bottleneck in
realistic cases. The benchmark used in this thread does many concurrent
sequential scans, on data that exceeds shared buffers / fits into RAM.
Perhaps that happens, but I doubt it's all that common.
I've been unable to demonstrate any benefits on other workloads, even if
there's a lot of buffer misses / reads into shared buffers. As soon as
the query starts doing something else, the clocksweep contention becomes
a non-issue. Consider for example read-only pgbench with database much
larger than shared buffers (but still within RAM). The cost of the index
scans (and other nodes) seems to reduce the pressure on clocksweep.
So I'm skeptical about clocksweep pressure being a serious issue, except
for some very narrow benchmarks (like the concurrent seqscan test). And
even if this happened for some realistic cases, partitioning the buffers
in a NUMA-oblivious way seems to do the trick.
When discussing this stuff off list, it was suggested this might help
with the scenario Andres presented in [3], where the throughput improves
a lot with multiple databases. I've not observed that in practice, and I
don't think these patches really can help with that. That scenario is
about buffer lock contention, not clocksweep contention.
With a single database there's nothing to do - there's one contended
page, located on a single node. There'll be contention, the no matter
which node it ends up on. With multiple databases (or multiple root
pages), it either happens to work by chance (if the buffers happen to be
from different nodes), or it would require figuring out which buffers
are busy, and place them on different nodes. But the patches did not
even try to do anything like that. So it still was a matter of chance.
That does not mean we don't need to worry about NUMA. There's still the
issue of misbalancing the allocation - with memory coming from just one
node, etc. Which is an issue because it "overloads" the memory subsystem
on that particular node. But that can be solved simply by interleaving
the shared memory segment.
That sidesteps most of the complexity - it does not need to figure out
how to partition buffers, does not need to worry about huge pages, etc.
This is not a new idea - it's more or less what Jakub Wartak initially
proposed in [2], before I hijacked the thread into the more complex
approach.
Attached is a tiny patch doing mostly what Jakub did, except that it
does two things. First, it allows interleaving the shared memory on all
relevant NUMA nodes (per numa_get_mems_allowed). Second, it allows
populating all memory by setting MAP_POPULATE in mmap(). There's a new
GUC to enable each of these.
I think we should try this (much simpler) approach first, or something
close to it. Sorry for dragging everyone into a much more complex
approach, which now seems to be a dead end.
regards
[1]
https://www.postgresql.org/message-id/[email protected]
[2]
https://www.postgresql.org/message-id/[email protected]...
[3] https://youtu.be/V75KpACdl6E?t=2120
--
Tomas Vondra
Attachments:
[text/x-patch] v20251208-0001-numa-Simple-interleaving-and-MAP_POPULATE.patch (8.9K, ../../[email protected]/2-v20251208-0001-numa-Simple-interleaving-and-MAP_POPULATE.patch)
download | inline diff:
From 72b0a9b86da7e1a6e239f5fdead24aeaa3d99f27 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Fri, 5 Dec 2025 14:53:47 +0100
Subject: [PATCH v20251208] numa: Simple interleaving and MAP_POPULATE
Allows NUMA interleaving on the shared memory segment, to ensure memory
is balanced between the NUMA nodes. The patch also allows prefaulting
the shared memory (by setting MAP_POPULATE flag), to actually allocate
the pages on nodes.
The commit addes two GUC parameters, both set to 'off' by default:
- shared_memory_interleave (enables NUMA interleaving)
- shared_memory_populate (sets MAP_POPULATE)
The memory is interleaved on all nodes enabled in the cpuset, as
returned by numa_get_mems_allowed(). The interleaving is applied at the
memory page granularity, and is oblivious to what's stored in it.
---
src/backend/port/sysv_shmem.c | 51 ++++++++++++++++++-
src/backend/utils/misc/guc_parameters.dat | 12 +++++
src/backend/utils/misc/guc_tables.c | 2 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/port/pg_numa.h | 4 ++
src/include/storage/pg_shmem.h | 3 ++
src/port/pg_numa.c | 42 +++++++++++++++
7 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 298ceb3e218..b300f5ef4a7 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include <numa.h>
#include <signal.h>
#include <unistd.h>
#include <sys/file.h>
@@ -29,6 +30,7 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
+#include "port/pg_numa.h"
#include "portability/mem.h"
#include "storage/dsm.h"
#include "storage/fd.h"
@@ -602,6 +604,28 @@ CreateAnonymousSegment(Size *size)
void *ptr = MAP_FAILED;
int mmap_errno = 0;
+ /*
+ * When asked to use NUMA-interleaving, we need to set the policy before
+ * touching the memory. By default that'll happen later, when either
+ * initializing some of the shmem structures (e.g. buffer descriptors), or
+ * when running queries. In that case it's enough to set the policy after
+ * the mmap() call, and we don't need to do anything here.
+ *
+ * With MAP_POPULATE, the mmap() itself will prefault the pages, so we
+ * need to set the policy to interleave before the mmap() call, and then
+ * revert to localalloc (so that private memory is allocated locally).
+ *
+ * XXX It probably is not a good idea to enable interleaving with regular
+ * memory pages, because then each buffer will get split on two nodes, and
+ * the system won't be able to fix that by migrating one of the pages. But
+ * we leave that up to the admin, instead of forbidding it.
+ */
+ if (shared_memory_interleave && shared_memory_populate)
+ {
+ /* set the allocation to interleave on nodes allowed by the cpuset */
+ pg_numa_set_interleave();
+ }
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -619,6 +643,10 @@ CreateAnonymousSegment(Size *size)
if (allocsize % hugepagesize != 0)
allocsize += hugepagesize - (allocsize % hugepagesize);
+ /* populate the shared memory if requested */
+ if (shared_memory_populate)
+ mmap_flags |= MAP_POPULATE;
+
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
PG_MMAP_FLAGS | mmap_flags, -1, 0);
mmap_errno = errno;
@@ -638,13 +666,19 @@ CreateAnonymousSegment(Size *size)
if (ptr == MAP_FAILED && huge_pages != HUGE_PAGES_ON)
{
+ int mmap_flags = 0;
+
+ /* populate the shared memory if requested */
+ if (shared_memory_populate)
+ mmap_flags |= MAP_POPULATE;
+
/*
* Use the original size, not the rounded-up value, when falling back
* to non-huge pages.
*/
allocsize = *size;
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS, -1, 0);
+ PG_MMAP_FLAGS | mmap_flags, -1, 0);
mmap_errno = errno;
}
@@ -663,6 +697,21 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+ /*
+ * With NUMA interleaving, we need to either apply interleaving for the
+ * shmem segment we just allocated, or reset the memory policy to local
+ * allocation (when using MAP_POPULATE).
+ */
+ if (shared_memory_interleave)
+ {
+ if (shared_memory_populate)
+ /* revert back to using the local node */
+ pg_numa_set_localalloc();
+ else
+ /* apply interleaving to the new memory segment */
+ pg_numa_interleave_memory(ptr, allocsize);
+ }
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 3b9d8349078..ce1c0c4327f 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2597,6 +2597,18 @@
max => 'INT_MAX / 2',
},
+{ name => 'shared_memory_interleave', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
+ short_desc => 'Enables NUMA interleaving of shared memory.',
+ variable => 'shared_memory_interleave',
+ boot_val => 'false',
+},
+
+{ name => 'shared_memory_populate', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
+ short_desc => 'Populates shared memory at start.',
+ variable => 'shared_memory_populate',
+ boot_val => 'false',
+},
+
{ name => 'shared_memory_size', type => 'int', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows the size of the server\'s main shared memory area (rounded up to the nearest MB).',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_UNIT_MB | GUC_RUNTIME_COMPUTED',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f87b558c2c6..cfee0df987f 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -580,6 +580,8 @@ static int ssl_renegotiation_limit;
int huge_pages = HUGE_PAGES_TRY;
int huge_page_size;
int huge_pages_status = HUGE_PAGES_UNKNOWN;
+bool shared_memory_interleave = false;
+bool shared_memory_populate = false;
/*
* These variables are all dummies that don't do anything, except in some
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index dc9e2255f8a..de1276f6897 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -152,6 +152,8 @@
# sysv
# windows
# (change requires restart)
+#shared_memory_interleave = off # interleave all memory on available NUMA nodes
+#shared_memory_populate = off # prefault shared memory on start
#dynamic_shared_memory_type = posix # the default is usually the first option
# supported by the operating system:
# posix
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..dc9f13d9fa0 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -18,6 +18,10 @@ extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT void pg_numa_set_interleave(void);
+extern PGDLLIMPORT void pg_numa_set_localalloc(void);
+extern PGDLLIMPORT void pg_numa_interleave_memory(void *ptr, Size size);
+
#ifdef USE_LIBNUMA
/*
diff --git a/src/include/storage/pg_shmem.h b/src/include/storage/pg_shmem.h
index 5f7d4b83a60..7b56bd5b44f 100644
--- a/src/include/storage/pg_shmem.h
+++ b/src/include/storage/pg_shmem.h
@@ -47,6 +47,9 @@ extern PGDLLIMPORT int huge_pages;
extern PGDLLIMPORT int huge_page_size;
extern PGDLLIMPORT int huge_pages_status;
+extern PGDLLIMPORT bool shared_memory_interleave;
+extern PGDLLIMPORT bool shared_memory_populate;
+
/* Possible values for huge_pages and huge_pages_status */
typedef enum
{
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 540ada3f8ef..a91d339033f 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -116,6 +116,33 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * Set allocation memory to interleave on all memory nodes in the cpuset.
+ */
+void
+pg_numa_set_interleave(void)
+{
+ numa_set_membind(numa_get_mems_allowed());
+}
+
+/*
+ * Set allocation memory to localalloc.
+ */
+void
+pg_numa_set_localalloc(void)
+{
+ numa_set_localalloc();
+}
+
+/*
+ * Set policy for memory to interleaving (on all nodes per cpuset).
+ */
+void
+pg_numa_interleave_memory(void *ptr, Size size)
+{
+ numa_interleave_memory(ptr, size, numa_get_mems_allowed());
+}
+
#else
/* Empty wrappers */
@@ -138,4 +165,19 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_set_interleave(void)
+{
+}
+
+void
+pg_numa_set_localalloc(void)
+{
+}
+
+void
+pg_numa_interleave_memory(void *ptr, Size size)
+{
+}
+
#endif
--
2.51.1
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-05 12:52 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2026-06-05 12:52 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Jakub Wartak <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Here's an updated version of the NUMA patch series, based on some recent
discussions about this (some at pgconf.dev, but not only that),
The main change is I significantly simplified some of the parts. Whe
patch from 20251126 was ~190K, the new version is maybe 100K, so about
half. Some of that is thanks to dropping the PGPROC partitioning
entirely, but the remaining parts are smaller too. I realize it's not a
great metric, of course.
In this message I'll explain the changes since 20251126. I'm yet to do a
thorough performance evaluation and see if it helps, I'll post that in
the next couple days.
The current patch series has these parts:
v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch
-------------
Somewhat unrelated, I find this useful for benchmarking and as a
baseline (what would happen if we just interleaved the shared segment).
v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch
-------------
Just adds a small registry of partitions (ranges of shared buffers),
stored in shared memory, and pg_buffercache interface to inspect it.
Merely a foundation for the following patches.
v20260605-0003-NUMA-shared-buffers-partitioning.patch
-------------
The interesting part, that places some of the partitions to NUMA nodes.
v20260605-0004-clock-sweep-basic-partitioning.patch
v20260605-0005-clock-sweep-balancing-of-allocations.patch
v20260605-0006-clock-sweep-scan-all-partitions.patch
-------------
Patches that gradually partition clock-sweep. Ultimately, it should
probably be squashed into a single commit (each commit fixes some sort
of issue in the naive partitioning in 0004). But I kept them separate
because it's easier to review / understand what the issue is.
what changed?
-------------
First, I dropped the PGPROC partitioning. We may revisit that in the
future (not sure), but for now it was just a distraction and I see it as
less impactful than shared buffers / clock-sweep.
I also simplified the GUC to use a single on/off parameter (instead of
the debug_io_direct-like approach). We can revisit that, but for now
this seems more convenient.
The most significant change in the remaining parts is simplification of
the shared buffer partitioning. In particular, the partitioning is now
"best-effort" when it maps memory to shared buffers. Let me remind that
NUMA works at memory page granularity - we can't map arbitrary ranges of
memory to a node, it needs to be whole memory pages.
The 20251126 patch went into great lengths to (a) make sure BufferBlocks
and BufferDescriptors start at memory page boundary, are the partitions
are also properly aligned (both for blocks and descriptors). That was a
lot of code, it needs to happen even before we know if huge pages are
used, partitions might have been of (very) different sizes, and so on.
The new patch abandons this "perfect" partitioning, and instead does a
best-effort. It splits the buffers as evenly as possible, i.e. all
partitions have (NBuffers/npartitions) buffers, and then locates as much
memory as possible to a selected NUMA node.
With 4K pages, that's always the whole partition. With huge pages (which
is expected of relevant NUMA systems), there may be a couple buffers at
the beginning/end of a partition. But it's less than one memory page,
per partition, and we expect the systems to have 10s or 100s of GBs, so
in the bigger scheme of things it's negligible (fractions of a percent).
For buffer descriptors the math is a bit worse - descriptors need much
less memory, but even there it should not be more than ~1%.
Seems perfectly fine to me. Or rather, the extra complexity does not
seem worth the possible benefit.
This also allowed dropping a part of the "clock-sweep partitioning"
patches, dealing with cases when the partitions are of different sizes.
With this new best-effort scheme the difference is at most 1 buffer, and
we can just ignore that.
questions
---------
At this point, my main question is whether there's a better way to
partition clock-sweep and/or do the balancing of allocations between
partitions. I believe it does work, but I have a feeling there might be
a more elegant way to do this kind of stuff (like an established
balancing algorithm of some sort).
The other thing I need to verify is how this behaves with
kernel.nr_hugepages. With some earlier versions it was easy to end in a
situation where everything seemed to work, but then much later the
kernel realized it does not have enough huge pages on a particular NUMA
node and crashed with a segfault (or was it sigbus?).
Of course, the other question is performance validation - does it even
help? I plan to repeat the various experiments mentioned in this thread
(by Andres and others) on available NUMA machines. But if someone has an
idea for another benchmark (and/or what metric to measure, not just the
usual duration), let me know.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20260605-0006-clock-sweep-scan-all-partitions.patch (6.2K, ../../[email protected]/2-v20260605-0006-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From 977831d85199b4b1b8d65de1131e4001459878fb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:39:38 +0200
Subject: [PATCH v20260605 6/6] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 83 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index a543fb12b21..1ac1e3e3490 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -184,6 +184,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint64 *buf_state);
/*
* clocksweep allocation balancing
@@ -251,10 +254,9 @@ static int clocksweep_partition_budget = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -486,7 +488,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -545,33 +548,61 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * a process "sweeps" only a fraction of buffers, even if the other buffers
- * are better candidates for eviction. Maybe there should be some logic to
- * "steal" buffers from other partitions or other nodes?
- *
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Needs to scan partitions if needed, as a fallback.
- *
- * XXX Would that also mean we should have multiple bgwriters, one for each
- * node, or would one bgwriter still handle all nodes?
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint64 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint64 old_buf_state;
uint64 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -599,7 +630,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -618,7 +649,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index f68e08e5697..ae977297849 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.54.0
[text/x-patch] v20260605-0005-clock-sweep-balancing-of-allocations.patch (27.4K, ../../[email protected]/3-v20260605-0005-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From ad46b08a3eaa018b562fdd4ca786c4c23d358ba5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:28:10 +0200
Subject: [PATCH v20260605 5/6] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 428 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 471 insertions(+), 21 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 92176fed7f8..43d2e84f9d2 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -20,7 +20,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 739c63b0cfc..c91f2bc5b4a 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -31,7 +33,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -889,6 +891,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -920,6 +924,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -941,11 +951,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -954,8 +970,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -984,6 +1008,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 02c75f82e5b..62e541abebd 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4135,6 +4135,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 2d56579682e..a543fb12b21 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -35,6 +35,26 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX We need to make ClockSweep fixed-size, so that we can have an array
+ * in shared memory. The easiest way is to pick a sufficiently high value
+ * that no system will actually need. 32 seems high enough.
+ *
+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average, so that we don't flap too much. The higher the value, the
+ * more the old value affects the result.
+ *
+ * XXX Doesn't this obscure the interpretation of weights as probabilities to
+ * allocate from a given partition? Does it still sum to 100%? I don't think
+ * so, it's just a fraction of allocations to go from a given partition.
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -68,9 +88,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that partition. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * Backends use the budget from it's "home" partition, so that a busy
+ * partitions (with a lot of processes on that NUMA node etc.) spread
+ * the allocations evenly.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -140,7 +183,66 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets a
+ * budget for allocating buffers from other partitions. A process that
+ * "exhausts" a budget in it's home partition gets redirected to the other
+ * partitions, driven by the budgets.
+ *
+ * For example, a partition may have budget [25, 25, 25, 25], which means
+ * each of the 4 partitions should get 1/4 of allocations. Or the buget
+ * can be [50, 50, 0, 0], which means all allocations will go to the first
+ * two partitions (one of them being the "home" one);
+ *
+ * We could do that based on a random number generator, but for now we
+ * simply treat the values as a budget, i.e. a number of allocations to
+ * serve from other partitions, and move in round-robin way.
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balanced. We can make the budgets higher
+ * (say, to match the expected number of allocations, i.e. bout the average
+ * number of allocations from the past interval). Or maybe configurable.
+ *
+ * XXX We should always start allocating from the "home" partition, i.e.
+ * from from it, and only then redirect to other partitions.
+ *
+ * XXX It probably is not great all the processes from that "home"
+ * partition are coordinated, and move to between partitions at about the
+ * same time. Not sure what to do about this.
+ *
+ * XXX We should also prefer other partitions from the same NUMA node (if
+ * there are some). Probably by setting the budgets.
+ *
+ * FIXME Explain at which point are the budgets recalculated, by which
+ * process, and how that affects other processes allocating buffers.
+ */
+
+/*
+ * The "optimal" clock-sweep partition. After a backend gets moved to a
+ * different NUMA node, we restart the balancing so that it uses the
+ * correct "budget" from the new home partition.
+ */
+static int clocksweep_partition_home = -1;
+
+/*
+ * The partition the backend is currently allocating from (either the
+ * home one, or one of the redirected ones).
+ */
+static int clocksweep_partition_current = -1;
+
+/*
+ * The number of buffers to allocate from the current partition.
+ */
+static int clocksweep_partition_budget = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -152,7 +254,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -300,11 +402,68 @@ ClockSweepPartitionIndex(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
+ /* What's the "optimal" partition for this backend? */
int index = ClockSweepPartitionIndex();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Was the process migrated to a different NUMA node? If the home partition
+ * changed, we need to reset the budget and start over, so that we correctly
+ * prefer "nearby" partitions etc.
+ *
+ * XXX Could this be a problem when processes move all the time? I don't
+ * think so - if a process moves between many partitions, that alone will
+ * spread the allocations over partitions. Similarly, if there are many
+ * processes, that should make it even more even.
+ */
+ if (clocksweep_partition_home != index)
+ {
+ clocksweep_partition_home = index;
+ clocksweep_partition_current = index;
+ clocksweep_partition_budget = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_home != -1);
+ Assert(clocksweep_partition_current != -1);
+ Assert(clocksweep_partition_budget >= 0);
+
+ /*
+ * When balancing allocations, redirect the allocations to other partitions
+ * according to the budgets. We move through partitions in a round-robin way,
+ * after allocating the "budget" of allocations from the current one.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of budget from the current partition? Move to the next one
+ * with non-zero budget.
+ */
+ while (clocksweep_partition_budget == 0)
+ {
+ /* wrap around at the end */
+ clocksweep_partition_current++;
+ if (clocksweep_partition_current >= StrategyControl->num_partitions)
+ clocksweep_partition_current = 0;
+
+ clocksweep_partition_budget
+ = sweep->balance[clocksweep_partition_current];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the current allocation */
+ --clocksweep_partition_budget;
+
+ /*
+ * Account for the allocation in the "home" partition, so that the next
+ * round of rebalancing (recalculating the budgets) knows about the
+ * allocation traffic in various partitions.
+ */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition_current];
}
/*
@@ -381,7 +540,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* between CPUs / NUMA nodes in between, these call may pick different
* partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -485,6 +644,229 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition budgets, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * This means an allocation may be requested in partition A (i.e. the
+ * home partition of the process requesting it), but end up allocating
+ * the buffer in partition B. We have a counter for both - the number of
+ * allocations requested in a partition, and the number of allocations
+ * actually handled by that partition. The former is used for calculating
+ * weights, the latter is used only for monitoring.
+ *
+ * The balancing happens in intervals - it adjusts future allocations
+ * based on stats about recent allocations, namely:
+ *
+ * - numBufferAllocs - number of allocations served by a partition
+ *
+ * - numRequestedAllocs - number of allocatios requested in a partition
+ *
+ * We're trying to smooth numBufferAllocs in the next interval, based on
+ * numRequestedAllocs measured in the last interval.
+ *
+ * The balancing algorithm works like this:
+ *
+ * - the target (average number of allocations per partition) is calculated
+ * from total number of allocations requested in the last intervaal
+ *
+ * - partitions get divided into two groups - those with more allocation
+ * requests than the target, and those with fewer requests
+ *
+ * - we "distribute" the delta (which is the same between the groups)
+ * between the groups (one has more, the other fewer)
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX Currently this does not give preference to other partitions on the
+ * same NUMA node (redirect to it first), but it could.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocation requests for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * XXX We lock the partitions one by one, so this is not a perfectly
+ * consistent snapshot of the counts, and the resets happen before we
+ * update the weights too. But we're only looking for heuristics, so
+ * this should be good enough.
+ *
+ * XXX A similar issue applies to the counter reset later - we haven't
+ * updated the weights yet, so some of the requests counted for the next
+ * interval will be redirected per current weights. Should be fine, it's
+ * just an approximate heuristics, and there should be very few requests in
+ * between. Alternatively, we could reset the request counters when setting
+ * the new weights, and just ignore the couple requests in between.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /* Calculate the "fair share" of allocations per partition. */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state for each partition, i.e. how
+ * many more/fewer allocations it handled relative to the average.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip rebalancing when there's not enough activity, and just keep the
+ * current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary. And maybe we should
+ * do the rabalancing in this case anyway, it's likely cheap and on a big
+ * system 10% can be quite a lot.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * The actual rebalancing
+ *
+ * Partition with fewer than average allocations, should not redirect any
+ * allocations to other partitions. So just use weights with a single
+ * non-zero weight for the partition itself.
+ *
+ * Partition with more than average allocations, should not receive any
+ * redirected allocations, and instead it should redirect excess allocations
+ * to other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of a
+ * partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from it.
+ *
+ * XXX We should add hysteresis, so that it does not oscillate or something
+ * like that. Maybe CLOCKSWEEP_HISTORY_COEFF already does that?
+ *
+ * XXX Ideally, the alternative partitions to use first would be the other
+ * partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -657,7 +1039,21 @@ StrategyCtlShmemInit(void *arg)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1025,8 +1421,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1034,11 +1432,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5ab0cee4281..887314d43f4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,6 +593,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e0bb4cc1df1..02833b19b0c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -411,11 +411,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.54.0
[text/x-patch] v20260605-0004-clock-sweep-basic-partitioning.patch (34.0K, ../../[email protected]/4-v20260605-0004-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 5ab7aba5f8af84b748180eb6caa2b165aceb2590 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:16:55 +0200
Subject: [PATCH v20260605 4/6] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 202 +++++++----
src/backend/storage/buffer/freelist.c | 333 ++++++++++++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 486 insertions(+), 104 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index a6e49fd1652..92176fed7f8 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -14,7 +14,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index e3efeeda675..739c63b0cfc 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -912,6 +912,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -933,12 +941,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -954,6 +972,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index cc398db124d..02c75f82e5b 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3826,33 +3826,34 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
- *
- * This is called periodically by the background writer process.
+ * Information saved between calls so we can determine the strategy point's
+ * advance rate and avoid scanning already-cleaned buffers.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX Does it actually make sense to split all of this information per
+ * partition? For example, does per-partition advance rate mean anything?
+ * Maybe we should have a global advance rate? Although, if we want to
+ * keep enough clean buffers in each partition, maybe having per-partition
+ * rates makes sense.
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+typedef struct BufferSyncPartition
+{
+ int prev_strategy_buf_id;
+ uint32 prev_strategy_passes;
+ int next_to_clean;
+ uint32 next_passes;
+} BufferSyncPartition;
+
+static BufferSyncPartition *saved_info = NULL;
+static bool saved_info_valid = false;
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition,
+ BufferSyncPartition *saved)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3880,25 +3881,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3910,17 +3902,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - saved->prev_strategy_passes;
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - saved->prev_strategy_buf_id;
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (saved->next_passes - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - saved->next_to_clean;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3928,11 +3920,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (saved->next_passes == strategy_passes &&
+ saved->next_to_clean >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (saved->next_to_clean - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3952,9 +3944,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3968,15 +3960,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ saved->prev_strategy_buf_id = strategy_buf_id;
+ saved->prev_strategy_passes = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3984,9 +3977,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3996,7 +3989,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -4004,10 +3997,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -4034,7 +4027,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -4059,20 +4052,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(saved->next_to_clean, true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++saved->next_to_clean >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ saved->next_to_clean = first_buffer;
+ saved->next_passes++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -4086,7 +4079,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -4116,8 +4109,83 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ /* allocate space for per-partition information between calls */
+ if (saved_info == NULL)
+ {
+ /*
+ * XXX Not great it's using malloc(), but how else to allocate a
+ * variable-length array?
+ */
+ saved_info = malloc(sizeof(BufferSyncPartition) * num_partitions);
+ }
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition,
+ &saved_info[partition]);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 53ef5239e8d..2d56579682e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -36,17 +36,28 @@
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock sweep partition */
+ int32 node;
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -57,11 +68,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+
+ /* cached info about freelist partitioning */
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -108,6 +140,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -119,6 +152,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -126,14 +160,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -159,19 +193,118 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /*
+ * Make sure we've calculated a buffer in the range of the partition. Buffer
+ * IDs are 1-based, we're calculating 0-based indexes.
+ */
+ Assert((victim >= 0) && (victim < sweep->numBuffers));
+ Assert(BufferIsValid(1 + sweep->firstBuffer + victim));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * ClockSweepPartitionIndex
+ * pick the clock-sweep partition to use based on PID and NUMA node
+ *
+ * With libnuma, use the NUMA node and PID to pick the partition. Otherwise
+ * use just PID (as if there's a single NUMA node).
+ *
+ * XXX This should also check if buffers are NUMA-partitioned, not just if
+ * compiled with libnuma.
+ */
+static int
+ClockSweepPartitionIndex(void)
+{
+ int node = 0,
+ index;
+ pid_t pid = MyProcPid;;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * If buffers are NUMA-partitioned, determine the partition using the NUMA
+ * node and PID. Without NUMA assume everything is a single NUMA node 0, and
+ * we pick the partition based on PID.
+ */
+#ifdef USE_LIBNUMA
+ if (shared_buffers_numa)
+ {
+ int cpu;
+
+ /* XXX do we need to check sched_getcpu is available, somehow? */
+ if ((cpu = sched_getcpu()) < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+ }
+#endif
+
+ /*
+ * We should't get unexpected NUMA nodes, not considered when setting up the
+ * buffer partitions. It could happen if the allowed NUMA nodes get adjusted
+ * at runtime, but at this point we just create partitions for all existing
+ * nodes. We could plan for allowed partitions, but then what if those get
+ * disabled, and the user allows some other partitions?
+ */
+ if ((node < 0) || (node > StrategyControl->num_nodes))
+ elog(ERROR, "node out of range: %d > %u", node, StrategyControl->num_nodes);
+
+ /*
+ * Calculate the partition index. Nodes have the same number of partitions,
+ * and we use the PID to pick one of those (for a given node). If there's
+ * only a single partition per node, we can ignore PID and use node directly.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = node;
+ }
+ else
+ {
+ /* use PID to pick one of node's partitions */
+ index = (node * StrategyControl->num_partitions_per_node)
+ + (pid % StrategyControl->num_partitions_per_node);
+ }
+
+ /* should have a valid partition index */
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and PID
+ *
+ * Pick a partition mapped to the NUMA node the backend is currently running
+ * on, and use PID if there are multiple partitions per node. Without NUMA
+ * supported/enabled, use just PID.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = ClockSweepPartitionIndex();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -242,10 +375,37 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* We count buffer allocation requests so that the bgwriter can estimate
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
+ *
+ * XXX It's not quite right we call ChooseClockSweep twice - now, and then
+ * a couple lines later (through ClockSweepTick). If the process moves
+ * between CPUs / NUMA nodes in between, these call may pick different
+ * partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * a process "sweeps" only a fraction of buffers, even if the other buffers
+ * are better candidates for eviction. Maybe there should be some logic to
+ * "steal" buffers from other partitions or other nodes?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Needs to scan partitions if needed, as a fallback.
+ *
+ * XXX Would that also mean we should have multiple bgwriters, one for each
+ * node, or would one bgwriter still handle all nodes?
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -325,6 +485,48 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* XXX Do we need the lock, if we're only accessing atomics? Surely not. */
+ /* XXX Are we ever calling this without num_buf_alloc? */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -332,37 +534,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -394,8 +603,14 @@ StrategyNotifyBgWriter(int bgwprocno)
static void
StrategyCtlShmemRequest(void *arg)
{
+ int num_partitions;
+
+ /* get the number of buffer partitions */
+ BufferPartitionsCalculate(NULL, &num_partitions, NULL);
+
ShmemRequestStruct(.name = "Buffer Strategy Status",
- .size = sizeof(BufferStrategyControl),
+ .size = offsetof(BufferStrategyControl, sweeps) +
+ mul_size(num_partitions, sizeof(ClockSweep)),
.ptr = (void **) &StrategyControl
);
}
@@ -408,12 +623,42 @@ StrategyCtlShmemInit(void *arg)
{
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Remember the number of partitions */
+ BufferPartitionsParams(&StrategyControl->num_nodes,
+ &StrategyControl->num_partitions,
+ &StrategyControl->num_partitions_per_node);
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].node = node;
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -777,3 +1022,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e944cee2e91..5ab0cee4281 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,7 +593,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
/* buf_table.c */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 1cf09e8fb7c..e0bb4cc1df1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -410,6 +410,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index ae977297849..f68e08e5697 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ea756015249..183570b4d43 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -451,6 +451,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.54.0
[text/x-patch] v20260605-0003-NUMA-shared-buffers-partitioning.patch (26.6K, ../../[email protected]/5-v20260605-0003-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 4672b0b53f68177ca76b9400e7f9ca4172a08ddb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 23:27:13 +0200
Subject: [PATCH v20260605 3/6] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc.
In cases like pre-warming a database from a single worker (e.g. using
pg_prewarm), we may end up with severely unbalanced memory distribution
(with most memory located on a single NUMA node). Unbalanced allocation
may put a lot of pressure on the memory system on a small number of NUMA
nodes, limiting the bandwidth etc.
With zone_reclaim, the kernel would eventually move some of the memory
to other nodes, but that tends to take a long time and is unpredictable.
This change forces even distribution of shared buffers on all NUMA
nodes, improving predictability, reducing the time needed for warmup
during benchmarking, etc. It's also less dependent on what the CPU
scheduler decides to do (which cores get used for the warmup.)
The effect is similar to
numactl --interleave=all
in that the buffers are distributed on the NUMA nodes evenly, but
there's also a number of important differences.
Firstly, it's applied only to shared buffers (and buffer descriptors),
not to the whole shared memory segment. It's possible to enable memory
interleaving using the shmem_interleave GUC, introduced in an earlier
patch in this series.
NUMA works at the granularity of a memory page, which is typically
either 4K or 2MB (hugepage), but other sizes are possible. For systems
where NUMA matters, we expect large amounts of memory (hundreds of
gigabytes) and hugepages enabled. But not necessarily.
The partitioning scheme is best-effort with respect to memory page size.
The shared buffers do not "align" with memory pages (i.e. a partition
may not end at the memory page boundary), in which case we simply locate
just the section of the partition with complete memory pages. This means
there may be ~one unmapped memory page between partitions. Considering
the expected amounts of memory, this is negligible, and the alternative
would be a significant amount of complexity to align the pages and
enforce "allowed" partition sizes.
Buffer descriptors are affected by this too, and the effect may be more
significant, simply because the descriptors are much smaller (~64B). So
the array is smaller, and a single 2MB memory page is worth ~32K buffer
descriptors. But with large systems it's still negligible.
The "buffer partitions" may not be 1:1 with NUMA nodes. We want to allow
clock-sweep partitioning even on non-NUMA systems, or when running only
on a small number of NUMA nodes. There's a minimal number of partitions
(default: 4), and a node may get multiple partitions. Nodes always get
the same number of partitions (e.g. with 3 NUMA nodes there will be 6
partitions in total, as each node gets 2 partitions).
The feature is enabled by dshared_buffers_numa GUC (default: false).
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/buf_init.c | 244 ++++++++++++++++--
src/backend/storage/buffer/freelist.c | 9 +
src/backend/utils/misc/guc_parameters.dat | 6 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/port/pg_numa.h | 7 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 8 +
src/port/pg_numa.c | 112 ++++++++
10 files changed, 392 insertions(+), 36 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index d62b8339bfc..a6e49fd1652 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer); -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index d678cb045ba..e3efeeda675 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -904,11 +904,13 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -926,28 +928,32 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index e593b02e0ca..1f93a31d451 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,12 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proclist.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -71,9 +79,12 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
-/* number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+bool shared_buffers_numa = false;
/*
* Register shared memory area for the buffer pool.
@@ -81,6 +92,10 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
static void
BufferManagerShmemRequest(void *arg)
{
+ int nparts;
+
+ BufferPartitionsCalculate(NULL, &nparts, NULL);
+
ShmemRequestStruct(.name = "Buffer Descriptors",
.size = NBuffers * sizeof(BufferDescPadded),
/* Align descriptors to a cacheline boundary. */
@@ -103,7 +118,7 @@ BufferManagerShmemRequest(void *arg)
);
ShmemRequestStruct(.name = "Buffer Partition Registry",
- .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ .size = nparts * sizeof(BufferPartition),
/* Align descriptors to a cacheline boundary. */
.alignment = PG_CACHE_LINE_SIZE,
.ptr = (void **) &BufferPartitionsRegistry,
@@ -134,6 +149,10 @@ BufferManagerShmemInit(void *arg)
/*
* Initialize the buffer partition registry first, before other parts
* have a chance to touch the memory.
+ *
+ * Also moves memory to different NUMA nodes (if enabled by a GUC).
+ * Do this before the loop that initializes buffer headers etc. which
+ * may fault some of the memory pages etc.
*/
BufferPartitionsInit();
@@ -231,35 +250,203 @@ BufferPartitionsInit(void)
{
int buffer = 0;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
- int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+ int nnodes,
+ npartitions,
+ npartitions_per_node;
- BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ int buffers_per_partition,
+ buffers_remaining;
- for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
- {
- BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+ /* calculate partitioning parameters */
+ BufferPartitionsCalculate(&nnodes, &npartitions, &npartitions_per_node);
+
+ /* paranoia */
+ Assert(nnodes > 0);
+ Assert(npartitions >= MIN_BUFFER_PARTITIONS);
+ Assert((npartitions % nnodes) == 0);
+ Assert((npartitions_per_node * nnodes) == npartitions);
- int num_buffers = part_buffers;
- if (n < remaining_buffers)
- num_buffers += 1;
+ BufferPartitionsRegistry->nnodes = nnodes;
+ BufferPartitionsRegistry->npartitions = npartitions;
+ BufferPartitionsRegistry->npartitions_per_node = npartitions_per_node;
- remaining_buffers -= num_buffers;
+ /* regular partition size, the first couple get an extra buffer */
+ buffers_per_partition = (NBuffers / npartitions);
+ buffers_remaining = (NBuffers % buffers_per_partition);
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ /* should have all the buffers */
+ Assert((buffers_per_partition * npartitions + buffers_remaining) == NBuffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ /*
+ * Now walk the partitions, and set the buffer range. Optionally, place
+ * the partitions on a given node (for all partitions at once).
+ */
+ for (int n = 0; n < nnodes; n++)
+ {
+ for (int p = 0; p < npartitions_per_node; p++)
+ {
+ int idx = (n * npartitions_per_node) + p;
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ /*
+ * Assign to the NUMA node, but only with shared_buffers_numa=on.
+ *
+ * XXX we should get an actual node ID from the mask, in case the
+ * task is restricted to only some nodes.
+ */
+ part->numa_node = (shared_buffers_numa) ? n : -1;
+
+ /* The first couple partitions may get an extra buffer. */
+ part->num_buffers = buffers_per_partition;
+ if (idx < buffers_remaining)
+ part->num_buffers += 1;
+
+ /* remember the buffer range */
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (part->num_buffers - 1);
+
+ /* remember start of the next partition */
+ buffer += part->num_buffers;
+ }
- buffer += num_buffers;
+#ifdef USE_LIBNUMA
+ /*
+ * Now try to locate buffers and buffer descriptors to the node (all
+ * partitions for the node at once).
+ */
+ if (shared_buffers_numa)
+ {
+ Size numa_page_size = pg_numa_page_size();
+
+ int part_first,
+ part_last,
+ buff_first,
+ buff_last;
+
+ char *startptr,
+ *endptr;
+
+ /* first/last partition for this node */
+ part_first = (n * npartitions_per_node);
+ part_last = part_first + (npartitions_per_node - 1);
+
+ /* buffers (blocks) */
+
+ /* first/last buffer */
+ buff_first = BufferPartitionsRegistry->partitions[part_first].first_buffer;
+ buff_last = BufferPartitionsRegistry->partitions[part_last].last_buffer;
+
+ /* beginning of the first block, end of last block */
+ startptr = BufferBlocks + ((Size) buff_first * BLCKSZ);
+ endptr = BufferBlocks + ((Size) (buff_last + 1) * BLCKSZ);
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers for node %d not well aligned [%p,%p] aligned [%p,%p]",
+ n, startptr, endptr,
+ (char *) TYPEALIGN(numa_page_size, startptr),
+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ pg_numa_bind_to_node(startptr, endptr, n);
+
+ /* buffer descriptors */
+
+ /* beginning of the first descriptor, end of last descriptor */
+ startptr = (char *) &BufferDescriptors[buff_first];
+ endptr = (char *) &BufferDescriptors[buff_last] + 1;
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers descriptors for node %d not well aligned [%p,%p] aligned [%p,%p]",
+ n, startptr, endptr,
+ (char *) TYPEALIGN(numa_page_size, startptr),
+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ pg_numa_bind_to_node(startptr, endptr, n);
+ }
+#endif
}
AssertCheckBufferPartitions();
}
+/*
+ * BufferPartitionsCalculate
+ * Pick number of buffer partitions for the number of nodes and
+ * MIN_BUFFER_PARTITIONS.
+ *
+ * Picks the smallest number of partitions higher thah MIN_BUFFER_PARTITIONS,
+ * such that all nodes have the same number of partitions.
+ *
+ * This is best-effort with respect to size of the partitions. It's possible
+ * the partitions are not a perfect multiple of page size, in which case
+ * we set location only for the part where that is possible. The buffers on
+ * the "boundary" may get located up on arbitrary nodes.
+ *
+ * The extra complexity of figuring out the right "partition size" is not
+ * worth it, and it can lead to some partitions being much smaller. This way
+ * we end up with partitions of almost exactly the same size (one BLCKSZ is
+ * the largest difference).
+ *
+ * We expect shared buffers to be much larger than page size (at least on
+ * system where NUMA is a relevant feature), so the number of "not located"
+ * buffers should be a negligible fraction. This only affects pages between
+ * partitions for different nodes, so (nodes-1) pages. This is certainly
+ * fine with 2MB huge pages, but even with 1GB pages it should be OK (as
+ * such systems should have humongous amounts of memory).
+ *
+ * It also means we don't need to worry about memory page size before knowing
+ * if huge pages got used (which we only learn during allocation).
+ */
+void
+BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ int nnodes,
+ nparts,
+ nparts_per_node;
+
+#if USE_LIBNUMA
+ nnodes = numa_num_configured_nodes();
+ nparts_per_node = 1; /* at least one partition per node */
+
+ while ((nparts_per_node * nnodes) < MIN_BUFFER_PARTITIONS)
+ nparts_per_node++;
+
+ nparts = (nnodes * nparts_per_node);
+#else
+ /* without NUMA, assume there's just one node */
+ nnodes = 1;
+ nparts = MIN_BUFFER_PARTITIONS;
+ nparts_per_node = MIN_BUFFER_PARTITIONS;
+#endif
+
+ if (num_nodes)
+ *num_nodes = nnodes;
+
+ if (num_partitions)
+ *num_partitions = nparts;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = nparts_per_node;
+}
+
/*
* BufferPartitionCount
* Returns the number of partitions created.
@@ -277,13 +464,14 @@ BufferPartitionCount(void)
* The returned information is first/last buffer, number of buffers.
*/
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
{
BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -293,3 +481,17 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+void
+BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ if (num_nodes)
+ *num_nodes = BufferPartitionsRegistry->nnodes;
+
+ if (num_partitions)
+ *num_partitions = BufferPartitionsRegistry->npartitions;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = BufferPartitionsRegistry->npartitions_per_node;
+}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index fdb5bad7910..53ef5239e8d 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,15 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f15e74198c5..2e71c04282c 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2724,6 +2724,12 @@
max => 'INT_MAX / 2',
},
+{ name => 'shared_buffers_numa', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
+ short_desc => 'Locate partitions of shared buffers (and descriptors) to NUMA nodes.',
+ variable => 'shared_buffers_numa',
+ boot_val => 'false',
+},
+
{ name => 'shared_memory_size', type => 'int', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows the size of the server\'s main shared memory area (rounded up to the nearest MB).',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_UNIT_MB | GUC_RUNTIME_COMPUTED',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..c0f79c779cc 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -142,6 +142,7 @@
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
# (change requires restart)
+#shared_buffers_numa = off # NUMA-aware partitioning
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB # min 64kB
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 1b668fe1d91..8fe4d4ab7e3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,13 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+extern PGDLLIMPORT int pg_numa_bind_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e5a887b9969..e944cee2e91 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -365,10 +365,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -378,7 +378,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -416,8 +416,12 @@ extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
extern int BufferPartitionCount(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
+extern void BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 79a3f44747a..1cf09e8fb7c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -158,10 +158,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -170,7 +172,9 @@ typedef struct BufferPartition
/* an array of information about all partitions */
typedef struct BufferPartitions
{
+ int nnodes; /* number of NUMA nodes */
int npartitions; /* number of partitions */
+ int npartitions_per_node; /* for convenience */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -206,6 +210,7 @@ extern PGDLLIMPORT const PgAioHandleCallbacks aio_local_buffer_readv_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
+extern PGDLLIMPORT bool shared_buffers_numa;
/* in localbuf.c */
extern PGDLLIMPORT int NLocBuffer;
@@ -390,6 +395,9 @@ extern void MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied,
int32 *buffers_already_dirty,
int32 *buffers_skipped);
+/* in buf_init.c */
+extern int BufferGetNode(Buffer buffer);
+
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 8954669273a..11c8a4503da 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -118,6 +121,94 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ int ret;
+ struct bitmask *nodemask;
+
+ if (node < 0)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ nodemask = numa_allocate_nodemask();
+ if (nodemask == NULL)
+ {
+ errno = ENOMEM;
+ return -1;
+ }
+
+ numa_bitmask_setbit(nodemask, node);
+
+ /*
+ * MPOL_BIND places the pages strictly on the node, and MPOL_MF_MOVE migrates
+ * pages already faulted in to that node. If mbind() fails, leave the default
+ * placement in effect, and report the failure.
+ */
+ ret = mbind(startptr, (endptr - startptr),
+ MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
+
+ numa_free_nodemask(nodemask);
+
+ return ret;
+}
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
#else
/* Empty wrappers */
@@ -140,4 +231,25 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+Size
+pg_numa_page_size(void)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
--
2.54.0
[text/x-patch] v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch (14.3K, ../../[email protected]/6-v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch)
download | inline diff:
From 052ac2256afbba3f25f20f683449a0bbb0af4241 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:09:33 +0200
Subject: [PATCH v20260605 2/6] Infrastructure for partitioning of shared
buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep or to make the
partitioning NUMA-aware).
The registry is a small array of BufferPartition entries in shared
memory, with partitions sized to be a fair share of shared buffers.
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
* The buffers are divided as evenly as possible, with the first couple
partitions possibly getting an extra buffer.
---
contrib/pg_buffercache/Makefile | 3 +-
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 142 ++++++++++++++++++
src/include/storage/buf_internals.h | 5 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 0e618f66aec..7fd5cdfc43d 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,8 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql \
+ pg_buffercache--1.7--1.8.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
new file mode 100644
index 00000000000..d62b8339bfc
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -0,0 +1,23 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index 11499550945..d2fa8ba53ba 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'
+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index bf2e6c97220..d678cb045ba 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,6 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -75,6 +76,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -871,3 +873,87 @@ pg_buffercache_mark_dirty_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of shared buffers.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 1407c930c56..e593b02e0ca 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -26,10 +26,12 @@ char *BufferBlocks;
ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsRegistry;
static void BufferManagerShmemRequest(void *arg);
static void BufferManagerShmemInit(void *arg);
static void BufferManagerShmemAttach(void *arg);
+static void BufferPartitionsInit(void);
const ShmemCallbacks BufferManagerShmemCallbacks = {
.request_fn = BufferManagerShmemRequest,
@@ -69,6 +71,9 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
+/* number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
/*
* Register shared memory area for the buffer pool.
@@ -97,6 +102,13 @@ BufferManagerShmemRequest(void *arg)
.ptr = (void **) &BufferIOCVArray,
);
+ ShmemRequestStruct(.name = "Buffer Partition Registry",
+ .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ /* Align descriptors to a cacheline boundary. */
+ .alignment = PG_CACHE_LINE_SIZE,
+ .ptr = (void **) &BufferPartitionsRegistry,
+ );
+
/*
* The array used to sort to-be-checkpointed buffer ids is located in
* shared memory, to avoid having to allocate significant amounts of
@@ -119,6 +131,12 @@ BufferManagerShmemRequest(void *arg)
static void
BufferManagerShmemInit(void *arg)
{
+ /*
+ * Initialize the buffer partition registry first, before other parts
+ * have a chance to touch the memory.
+ */
+ BufferPartitionsInit();
+
/*
* Initialize all the buffer headers.
*/
@@ -151,3 +169,127 @@ BufferManagerShmemAttach(void *arg)
WritebackContextInit(&BackendWritebackContext,
&backend_flush_after);
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsRegistry->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsRegistry->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsRegistry->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsRegistry->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * BufferPartitionsInit
+ * Initialize registry of buffer partitions.
+ */
+static void
+BufferPartitionsInit(void)
+{
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
+ int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+
+ int num_buffers = part_buffers;
+ if (n < remaining_buffers)
+ num_buffers += 1;
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+/*
+ * BufferPartitionCount
+ * Returns the number of partitions created.
+ */
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsRegistry->npartitions;
+}
+
+/*
+ * BufferPartitionGet
+ * Returns information about a partition at the provided index.
+ *
+ * The returned information is first/last buffer, number of buffers.
+ */
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 89615a254a3..e5a887b9969 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -411,9 +411,14 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsRegistry;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 6837b35fc6d..79a3f44747a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -155,6 +155,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8cf40c87043..ea756015249 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -365,6 +365,8 @@ BufferHeapTupleTableSlot
BufferLockMode
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.54.0
[text/x-patch] v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch (4.9K, ../../[email protected]/7-v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch)
download | inline diff:
From 238935f06c793dcd0271423af83e039d2ee00120 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 3 Jun 2026 16:30:28 +0200
Subject: [PATCH v20260605 1/6] Add shmem_populate and shmem_interleave GUCs
- shmem_populate - Forces mmap() with MAP_POPULATE, which faults all
memory pages backing the shared memory segment.
- shmem_interleave - Applies NUMA interleaving on the whole shared
memory segment, to balance allocations between nodes.
---
src/backend/port/sysv_shmem.c | 47 +++++++++++++++++++++++
src/backend/utils/misc/guc_parameters.dat | 14 +++++++
src/include/miscadmin.h | 4 ++
3 files changed, 65 insertions(+)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 2e3886cf9fe..9eaff838a04 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -27,6 +27,10 @@
#include <sys/shm.h>
#include <sys/stat.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "portability/mem.h"
@@ -98,6 +102,10 @@ void *UsedShmemSegAddr = NULL;
static Size AnonymousShmemSize;
static void *AnonymousShmem = NULL;
+/* GUCs */
+bool shmem_populate = false; /* MAP_POPULATE */
+bool shmem_interleave = false; /* NUMA interleaving */
+
static void *InternalIpcMemoryCreate(IpcMemoryKey memKey, Size size);
static void IpcMemoryDetach(int status, Datum shmaddr);
static void IpcMemoryDelete(int status, Datum shmId);
@@ -604,6 +612,21 @@ CreateAnonymousSegment(Size *size)
int mmap_errno = 0;
int mmap_flags = MAP_SHARED | MAP_ANONYMOUS | MAP_HASSEMAPHORE;
+ /* If requested, populate the shared memory by MAP_POPULATE. */
+ if (shmem_populate)
+ mmap_flags |= MAP_POPULATE;
+
+#ifdef USE_LIBNUMA
+ /*
+ * If requested, interleave the shared memory by setting a memory policy
+ * before the mmap() call. This really matters only with MAP_POPULATE,
+ * because without page faults the memory does not actually get placed
+ * to the nodes. But without MAP_POPULATE it's virtually free.
+ */
+ if (shmem_interleave)
+ numa_set_interleave_mask(numa_all_nodes_ptr);
+#endif
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -665,6 +688,30 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+#ifdef USE_LIBNUMA
+ /*
+ * If set the policy to interleaving by numa_set_membind(), undo it now by
+ * setting the policy to localalloc. With MAP_POPULATE, all the pages were
+ * faulted and are now interleaved on the available nodes.
+ *
+ * To handle the case without MAP_POPULATE, apply the interleaving policy to
+ * the shared memory segment allocated by mmap() before touching it in any
+ * way, so that it gets placed on the correct node on first access.
+ *
+ * This matters especially with huge pages, where it's possible to run out
+ * of huge pages on some nodes and then crash. By explicitly interleaving
+ * the whole segment, that's much less likely.
+ */
+ if (shmem_interleave)
+ {
+ /* undo the policy set by numa_set_membind() earlier */
+ numa_set_localalloc();
+
+ /* set interleaving policy for not yet faulted memory */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+ }
+#endif
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index afaa058b046..f15e74198c5 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -740,6 +740,20 @@
ifdef => 'DEBUG_NODE_TESTS_ENABLED',
},
+{ name => 'debug_shmem_interleave', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Forces interleaving for the whole shared memory segment.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_interleave',
+ boot_val => 'false'
+},
+
+{ name => 'debug_shmem_populate', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Populates (faults) the whole shared memory segment using MAP_POPULATE.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_populate',
+ boot_val => 'false'
+},
+
{ name => 'debug_write_read_parse_plan_trees', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Set this to force all parse and plan trees to be passed through outfuncs.c/readfuncs.c, to facilitate catching errors and omissions in those modules.',
flags => 'GUC_NOT_IN_SAMPLE',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 7de0a115402..13bb29f8702 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -213,6 +213,10 @@ extern PGDLLIMPORT Oid MyDatabaseTableSpace;
extern PGDLLIMPORT bool MyDatabaseHasLoginEventTriggers;
+extern PGDLLIMPORT bool shmem_populate;
+extern PGDLLIMPORT bool shmem_interleave;
+
+
/*
* Date/Time Configuration
*
--
2.54.0
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-16 08:16 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2026-06-16 08:16 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <[email protected]> wrote:
>
> Hi,
Hi Tomas, thanks for working on this.
> Here's an updated version of the NUMA patch series, based on some recent
> discussions about this (some at pgconf.dev, but not only that),
[..]
1. 005 says:
+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 32
but there isn't direct any check for checking if pg_numa_get_max_node() ->
numa_max_node() is not getting higher than allowed here. In theory this could
happen I think if ClockSweepPartitionIndex() would return
numa = numa_node_of_cpu()
on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
and that would cause accesing .balance[] without bounds.
2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
this be padded/aligned somehow later in BufferStrategyControl which does
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
to avoid contention/false sharing? (comments says it should be but it
doesn't seem so?), maybe the comment should be TODO for now? I have not
quantified any potential benefit
With pahole after some hassle I've got:
struct ClockSweep {
slock_t clock_sweep_lock; /* 0 1 */
/* XXX 3 bytes hole, try to pack */
int32 node; /* 4 4 */
int32 firstBuffer; /* 8 4 */
int32 numBuffers; /* 12 4 */
pg_atomic_uint32 nextVictimBuffer; /* 16 4 */
uint32 completePasses; /* 20 4 */
pg_atomic_uint32 numBufferAllocs; /* 24 4 */
pg_atomic_uint32 numRequestedAllocs; /* 28 4 */
pg_atomic_uint64 numTotalAllocs; /* 32 8 */
pg_atomic_uint64 numTotalRequestedAllocs; /* 40 8 */
uint8 balance[32]; /* 48 32 */
/* size: 80, cachelines: 2, members: 11 */
/* sum members: 77, holes: 1, sum holes: 3 */
/* last cacheline: 16 bytes */
};
maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?
3. In 004 sched_getcpu() is used and mentioned how to check if it is available
But my $0.02 (maybe not that important), but I've at least saw once where
(on EC2?) some clock_gettime() was very slow and that was because it was not
available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
-> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
available, but slow and it would mean real syscall price (and that's not once
there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
mind, but I haven't checked further). The point is: wouldn't it be cheaper
that to be refreshed from time to time instead otherwise we risk some slow
code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
Or alternative is to have pg_test_numa proggie and this would be measuring
certain things about NUMA including timing of sched_getcpu (just like
pg_test_timing does for time), at least that could explain why somebody's
system/platform is slow.
4. Patch has problem (without fix for #8) that when number of available huge
pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
will use only first NUMA node. This might be a problem when starting mulitple
DBs (they will occupy first available NUMA):
### with s_b=8GB and nr_hugepages=1500 it's OK
# find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
\; | grep 2048 | sort
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250
## note the correct split below for N0/N1..
# grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
# find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
\; | grep 2048 | sort
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
## all on N0...
# grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=4269 kernelpagesize_kB=2048
I was even thinking go to lengths and add code for inspecting that /sys on
some later date that the kernel NUMA hugepages are really distributed
on the nodes as they should be (it's easy to end up on just 1 node out of
many; allocating via sysctl -w <higher> and then <lower> allocation is easy
way to force hugepages just to 1 node instead of many :o). I've hit the
problem multiple times, so we should bail out if we want NUMA and the
Buffer Blocks were just put on 1 node (instead of many).
5. In 005 we could mention more clealry what's the difference between
those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
in the defintion to make it easier to read, maybe copy-cat those earlier
descriptions there too as we already have:
+ * The balancing happens in intervals - it adjusts future allocations
+ * based on stats about recent allocations, namely:
+ *
+ * - numBufferAllocs - number of allocations served by a partition
+ *
+ * - numRequestedAllocs - number of allocatios requested in a partition
6. While at it, it would be helpful if we could reset the
pg_buffercache_partitions stats in some way (pg_buffercache_partitions
is very usefull).. or is there way to plug into main pg_reset functions?
7. If I add basic error checking for mbind() then it complains a lot, like
below with annotated strace -ffe mbind to show the point:
[pid 2856] mbind(0x7fd8d0e00000, 2145386496, MPOL_BIND, [], 0,
MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING: mbind(): Invalid argument
WARNING: buffers descriptors for node 0 not well aligned
[0x7fd8cccf5000,0x7fd8cdcf4fc1] aligned
[0x7fd8cce00000,0x7fd8cdc00000]
[pid 2856] mbind(0x7fd8cce00000, 14680064, MPOL_BIND, [], 0,
MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING: mbind(): Invalid argument
WARNING: buffers for node 1 not well aligned
[0x7fd950cf5000,0x7fd9d0cf5000] aligned
[0x7fd950e00000,0x7fd9d0c00000]
[pid 2856] mbind(0x7fd950e00000, 2145386496, MPOL_BIND,
0x5589057ded00, 1, MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING: mbind(): Invalid argument
WARNING: buffers descriptors for node 1 not well aligned
[0x7fd8cdcf5000,0x7fd8cecf4fc1] aligned
[0x7fd8cde00000,0x7fd8cec00000]
[..]
but with pg_numa.c fixed like below (node should be size):
ret = mbind(startptr, (endptr - startptr),
- MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
+ MPOL_BIND, nodemask->maskp,
nodemask->size, MPOL_MF_MOVE);
it doesn't report errors anymore and suprisngly hugepages in numa_maps are
altered from:
7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
to explicit "binds":
7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
mapmax=6 N0=25 kernelpagesize_kB=2048
7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=3 N0=7 kernelpagesize_kB=2048
7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N1=7 kernelpagesize_kB=2048
7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N2=7 kernelpagesize_kB=2048
7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N3=7 kernelpagesize_kB=2048
7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=3 N0=1 kernelpagesize_kB=2048
7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
mapmax=2 N0=1023 kernelpagesize_kB=2048
7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N0=1 kernelpagesize_kB=2048
7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
N1=1023 kernelpagesize_kB=2048
7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N1=1 kernelpagesize_kB=2048
7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
N2=1023 kernelpagesize_kB=2048
7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048
7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
N3=1023 kernelpagesize_kB=2048
7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
mapmax=6 N2=117 kernelpagesize_kB=2048
so lots of VMAs were created (it could affect performance in some way, I think
for sure it would affect for worse fork() rates by postmaster for new conns).
To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
single HP page being somehow left/missed in address calculations, but I
haven't pressed this harder.
NOTE: the patch works even without this fix, but I believe if got non-0 we
cannot reliably trust the optimizer memory layout has been deployed (I suspect
it's some kind luck it sharded the shm based on number of hugepages available)
> questions
> ---------
>
> At this point, my main question is whether there's a better way to
> partition clock-sweep and/or do the balancing of allocations between
> partitions. I believe it does work, but I have a feeling there might be
> a more elegant way to do this kind of stuff (like an established
> balancing algorithm of some sort).
8. The crux of this email and stuff I wanted to further discuss, when server
is started with on this 4-NUMA box with
* numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
on node#0
* the shm split onto 4 nodes properly
* s_b still just 8GB (with ideal split),
* DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
* pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
\set num (:client_id % 8) + 1
select sum(octet_length(filler)) from pgbench_accounts_:num;
* mpstat repors correctly just node#0 used
a. with the patch for GUCs with numa on and defaults two clocksweep settings
on, I'm getting:
latency average = 3252.254 ms
latency stddev = 72.011 ms
b. with debug_clocksweep_balance=off, I'm realiably getting
latency average = 2688.742 ms
latency stddev = 61.738 ms
so IMHO clocksweep partitioning is cool, but if we are discussing the current
balancing logic leaves some juice on the table from the most optimized variant
(~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).
Dunno if it should be optimized further, certainly we'll get reports from
quick benchmarks run by people that PG 20 could be *slower* because.. well,
they got (sub)optimal layout during startup (all HP on 1 node and some
query hitting just that one query with local affinity and this is visible
to naked eye). I was re-reading thread and Andres also wrote "We should use
the partitioned clock sweep to default to using local memory as long as
possible."
So two further ideas:
I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
itself, couldn't we? If we are using capped ring buffer, we could indicate
that we want it just from local node as priority disregarding weights (?).
Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
there would be some potential issue with sync-scan-table code though.
With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
for ideal distribution amongst all NUMA nodes.
II. what if we could track if the relation is just all-local-access?
Another idea is that if we would know that's it's just us working on some
relation (created by us; or it's not being touched remotley) then we could
ask for local-only memory affinity. So something like this:
a. in case of locally-only access rels =>
ask for local memory first
if that fails failback to weighted RR (so to to weights)
b. in case other rels => weighted RR (so to to weights) directly
The tracking of the fact that Buffer was accessed just locally or remotley
itself is not hard to imagine (by using some free "bits" in BufferDesc.
"state" where refcount/usagecount itself are stored, well at least 4 bits
for my 4 nodes, but there's plenty of left there), I have some PoC for
that but that's just per-Buffer tracking of "was this Buffer accessed by
remote nodes", but I'm completley lost how to make transition to the
is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
optimization (we would need some shared infra just for tracking such info;
assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
2..4 bits, that's already huge number: we are talking GBs of shm mem).
BTW: I've been experimenting with this patchset and added couple of things
(see attached), and with I'm able to get optimal just by forcing affinity
too using that earlier bench:
latency average = 2512.929 ms
latency stddev = 97.525 ms
and that was with pure 100% affinity to my local node:
select pg_buffercache_set_partition(0, '{100,0,0,0}');
debug_clocksweep_balance_recalc=off
debug_clocksweep_balance=on
debug_clocksweep_scan_all_partitions=on
(so it's another proof that code is fine, it's just algorithm that would
have to adjusted)
For benchmarks with pgbench -S for 100% local affinity vs 100% remote
(I can do that with that pg_buffercache_set_partition() of mine), I'm
getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
I've spotted some another bug in from where we are fetching memory from
unoptimal places if we are not on node#0, I'll need to dig into that more
though. Another thing is that pgbench -S runs are much less demanding in
terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
seqconcurrscans.sql using the same amout of cores).
> The other thing I need to verify is how this behaves with
> kernel.nr_hugepages. With some earlier versions it was easy to end in a
> situation where everything seemed to work, but then much later the
> kernel realized it does not have enough huge pages on a particular NUMA
> node and crashed with a segfault (or was it sigbus?).
It was SIGBUS and with this patchset I think we are fine: I have never
witnessed this one crashing with SIGBUS.
> Of course, the other question is performance validation - does it even
> help? I plan to repeat the various experiments mentioned in this thread
> (by Andres and others) on available NUMA machines. But if someone has an
> idea for another benchmark (and/or what metric to measure, not just the
> usual duration), let me know.
See above, but I think we would have to fix at at least: mbind() failure,
and those VMAs disconnected regions.
-J.
Attachments:
[application/octet-stream] vXXX1-0001-Add-parttioned-clocksweep-and-NUMA-goodies.cfbotignorepatch (14.6K, ../../CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay+mOEKs9rzCkegUA@mail.gmail.com/2-vXXX1-0001-Add-parttioned-clocksweep-and-NUMA-goodies.cfbotignorepatch)
download
[application/octet-stream] mbind_check_errcode.cfbotignorepatch (871B, ../../CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay+mOEKs9rzCkegUA@mail.gmail.com/3-mbind_check_errcode.cfbotignorepatch)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-16 12:39 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2026-06-16 12:39 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On 6/16/26 10:16, Jakub Wartak wrote:
> On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <[email protected]> wrote:
>>
>> Hi,
>
> Hi Tomas, thanks for working on this.
>
>> Here's an updated version of the NUMA patch series, based on some recent
>> discussions about this (some at pgconf.dev, but not only that),
> [..]
>
> 1. 005 says:
>
> + * XXX We should enforce this in bufmgr.c, when initializing the partitions.
> + */
> +#define MAX_BUFFER_PARTITIONS 32
>
> but there isn't direct any check for checking if pg_numa_get_max_node() ->
> numa_max_node() is not getting higher than allowed here. In theory this could
> happen I think if ClockSweepPartitionIndex() would return
> numa = numa_node_of_cpu()
> on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
> and that would cause accesing .balance[] without bounds.
>
Yes, this should be capped to the MAX_BUFFER_PARTITIONS.
> 2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
> this be padded/aligned somehow later in BufferStrategyControl which does
> ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
> to avoid contention/false sharing? (comments says it should be but it
> doesn't seem so?), maybe the comment should be TODO for now? I have not
> quantified any potential benefit
>
> With pahole after some hassle I've got:
> struct ClockSweep {
> slock_t clock_sweep_lock; /* 0 1 */
>
> /* XXX 3 bytes hole, try to pack */
>
> int32 node; /* 4 4 */
> int32 firstBuffer; /* 8 4 */
> int32 numBuffers; /* 12 4 */
> pg_atomic_uint32 nextVictimBuffer; /* 16 4 */
> uint32 completePasses; /* 20 4 */
> pg_atomic_uint32 numBufferAllocs; /* 24 4 */
> pg_atomic_uint32 numRequestedAllocs; /* 28 4 */
> pg_atomic_uint64 numTotalAllocs; /* 32 8 */
> pg_atomic_uint64 numTotalRequestedAllocs; /* 40 8 */
> uint8 balance[32]; /* 48 32 */
>
> /* size: 80, cachelines: 2, members: 11 */
> /* sum members: 77, holes: 1, sum holes: 3 */
> /* last cacheline: 16 bytes */
> };
> maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?
>
Possibly. Im not entirely happy with making the ClockSweep struct so
much larger, but I haven't found a better way to store the counters
needed for balancing. The only thing I can think of is storing it
outside the struct, and maybe that's the right thing to do.
But that assumes the current balancing approach is the right one.
> 3. In 004 sched_getcpu() is used and mentioned how to check if it is available
>
> But my $0.02 (maybe not that important), but I've at least saw once where
> (on EC2?) some clock_gettime() was very slow and that was because it was not
> available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
> always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
> -> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
> available, but slow and it would mean real syscall price (and that's not once
> there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
> mind, but I haven't checked further). The point is: wouldn't it be cheaper
> that to be refreshed from time to time instead otherwise we risk some slow
> code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
> Or alternative is to have pg_test_numa proggie and this would be measuring
> certain things about NUMA including timing of sched_getcpu (just like
> pg_test_timing does for time), at least that could explain why somebody's
> system/platform is slow.
>
Yes, I think we may need some sort of caching for this / check only
sometimes. I haven't seen it to matter, but that may be luck and on
other systems / platforms it may be worse.
> 4. Patch has problem (without fix for #8) that when number of available huge
> pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
> will use only first NUMA node. This might be a problem when starting mulitple
> DBs (they will occupy first available NUMA):
>
> ### with s_b=8GB and nr_hugepages=1500 it's OK
>
> # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> \; | grep 2048 | sort
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250
>
> ## note the correct split below for N0/N1..
> # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
>
> ### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
> # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> \; | grep 2048 | sort
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
> ## all on N0...
> # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> 7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=4269 kernelpagesize_kB=2048
>
> I was even thinking go to lengths and add code for inspecting that /sys on
> some later date that the kernel NUMA hugepages are really distributed
> on the nodes as they should be (it's easy to end up on just 1 node out of
> many; allocating via sysctl -w <higher> and then <lower> allocation is easy
> way to force hugepages just to 1 node instead of many :o). I've hit the
> problem multiple times, so we should bail out if we want NUMA and the
> Buffer Blocks were just put on 1 node (instead of many).
>
How come the pg_numa_bind_to_node() calls don't move the parts to the
correct node?
If something is already using huge pages on the other nodes, then sure,
it will fail. But I think that's OK - it's a best-effort thing. Maybe we
should exit instead in this case?
> 5. In 005 we could mention more clealry what's the difference between
> those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
> in the defintion to make it easier to read, maybe copy-cat those earlier
> descriptions there too as we already have:
>
> + * The balancing happens in intervals - it adjusts future allocations
> + * based on stats about recent allocations, namely:
> + *
> + * - numBufferAllocs - number of allocations served by a partition
> + *
> + * - numRequestedAllocs - number of allocatios requested in a partition
>
I agree the explanation for this is not entirely clear.
> 6. While at it, it would be helpful if we could reset the
> pg_buffercache_partitions stats in some way (pg_buffercache_partitions
> is very usefull).. or is there way to plug into main pg_reset functions?
>
Hmm, I was afraid it'd interfere with the balancing. I'm not sure it
makes sense to reset just some of the fields - it'd make it much harder
to interpret the counters. I'll think abou this.
> 7. If I add basic error checking for mbind() then it complains a lot, like
> below with annotated strace -ffe mbind to show the point:
>
> [pid 2856] mbind(0x7fd8d0e00000, 2145386496, MPOL_BIND, [], 0,
> MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING: mbind(): Invalid argument
> WARNING: buffers descriptors for node 0 not well aligned
> [0x7fd8cccf5000,0x7fd8cdcf4fc1] aligned
> [0x7fd8cce00000,0x7fd8cdc00000]
>
> [pid 2856] mbind(0x7fd8cce00000, 14680064, MPOL_BIND, [], 0,
> MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING: mbind(): Invalid argument
> WARNING: buffers for node 1 not well aligned
> [0x7fd950cf5000,0x7fd9d0cf5000] aligned
> [0x7fd950e00000,0x7fd9d0c00000]
>
> [pid 2856] mbind(0x7fd950e00000, 2145386496, MPOL_BIND,
> 0x5589057ded00, 1, MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING: mbind(): Invalid argument
> WARNING: buffers descriptors for node 1 not well aligned
> [0x7fd8cdcf5000,0x7fd8cecf4fc1] aligned
> [0x7fd8cde00000,0x7fd8cec00000]
> [..]
>
> but with pg_numa.c fixed like below (node should be size):
> ret = mbind(startptr, (endptr - startptr),
> - MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
> + MPOL_BIND, nodemask->maskp,
> nodemask->size, MPOL_MF_MOVE);
>
I think this is a silly bug on my side, clearly the nodemask should have
size > 0 even for node 0.
> it doesn't report errors anymore and suprisngly hugepages in numa_maps are
> altered from:
> 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
>
> to explicit "binds":
> 7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
> mapmax=6 N0=25 kernelpagesize_kB=2048
> 7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=3 N0=7 kernelpagesize_kB=2048
> 7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N1=7 kernelpagesize_kB=2048
> 7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N2=7 kernelpagesize_kB=2048
> 7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N3=7 kernelpagesize_kB=2048
> 7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=3 N0=1 kernelpagesize_kB=2048
> 7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
> mapmax=2 N0=1023 kernelpagesize_kB=2048
> 7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N0=1 kernelpagesize_kB=2048
> 7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
> N1=1023 kernelpagesize_kB=2048
> 7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N1=1 kernelpagesize_kB=2048
> 7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
> N2=1023 kernelpagesize_kB=2048
> 7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N3=1 kernelpagesize_kB=2048
> 7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
> N3=1023 kernelpagesize_kB=2048
> 7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
> mapmax=6 N2=117 kernelpagesize_kB=2048
>
> so lots of VMAs were created (it could affect performance in some way, I think
> for sure it would affect for worse fork() rates by postmaster for new conns).
>
Isn't this 4 VMAs for buffers ad 4 VMAs for buffer descriptors? Or maybe
it's the parts that could not be mapped due to insufficient alignment?
> To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
> single HP page being somehow left/missed in address calculations, but I
> haven't pressed this harder.
>
> NOTE: the patch works even without this fix, but I believe if got non-0 we
> cannot reliably trust the optimizer memory layout has been deployed (I suspect
> it's some kind luck it sharded the shm based on number of hugepages available)
>
I'm not sure I understand what you mean. What non-0?
>
>> questions
>> ---------
>>
>> At this point, my main question is whether there's a better way to
>> partition clock-sweep and/or do the balancing of allocations between
>> partitions. I believe it does work, but I have a feeling there might be
>> a more elegant way to do this kind of stuff (like an established
>> balancing algorithm of some sort).
>
>
> 8. The crux of this email and stuff I wanted to further discuss, when server
> is started with on this 4-NUMA box with
> * numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
> on node#0
> * the shm split onto 4 nodes properly
> * s_b still just 8GB (with ideal split),
> * DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
> * pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
> \set num (:client_id % 8) + 1
> select sum(octet_length(filler)) from pgbench_accounts_:num;
> * mpstat repors correctly just node#0 used
>
> a. with the patch for GUCs with numa on and defaults two clocksweep settings
> on, I'm getting:
>
> latency average = 3252.254 ms
> latency stddev = 72.011 ms
>
> b. with debug_clocksweep_balance=off, I'm realiably getting
>
> latency average = 2688.742 ms
> latency stddev = 61.738 ms
>
> so IMHO clocksweep partitioning is cool, but if we are discussing the current
> balancing logic leaves some juice on the table from the most optimized variant
> (~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
> was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).
>
How does this compare to master, i.e. without these NUMA patches?
> Dunno if it should be optimized further, certainly we'll get reports from
> quick benchmarks run by people that PG 20 could be *slower* because.. well,
> they got (sub)optimal layout during startup (all HP on 1 node and some
> query hitting just that one query with local affinity and this is visible
> to naked eye). I was re-reading thread and Andres also wrote "We should use
> the partitioned clock sweep to default to using local memory as long as
> possible."
>
> So two further ideas:
>
> I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
> itself, couldn't we? If we are using capped ring buffer, we could indicate
> that we want it just from local node as priority disregarding weights (?).
> Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
> there would be some potential issue with sync-scan-table code though.
> With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
> prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
> for ideal distribution amongst all NUMA nodes.
>
Yes, I think it might make sense to disable balancing in these cases.
> II. what if we could track if the relation is just all-local-access?
>
> Another idea is that if we would know that's it's just us working on some
> relation (created by us; or it's not being touched remotley) then we could
> ask for local-only memory affinity. So something like this:
>
> a. in case of locally-only access rels =>
> ask for local memory first
> if that fails failback to weighted RR (so to to weights)
> b. in case other rels => weighted RR (so to to weights) directly
>
> The tracking of the fact that Buffer was accessed just locally or remotley
> itself is not hard to imagine (by using some free "bits" in BufferDesc.
> "state" where refcount/usagecount itself are stored, well at least 4 bits
> for my 4 nodes, but there's plenty of left there), I have some PoC for
> that but that's just per-Buffer tracking of "was this Buffer accessed by
> remote nodes", but I'm completley lost how to make transition to the
> is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
> optimization (we would need some shared infra just for tracking such info;
> assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
> 2..4 bits, that's already huge number: we are talking GBs of shm mem).
>> BTW: I've been experimenting with this patchset and added couple of things
> (see attached), and with I'm able to get optimal just by forcing affinity
> too using that earlier bench:
>
> latency average = 2512.929 ms
> latency stddev = 97.525 ms
>
> and that was with pure 100% affinity to my local node:
>
> select pg_buffercache_set_partition(0, '{100,0,0,0}');
> debug_clocksweep_balance_recalc=off
> debug_clocksweep_balance=on
> debug_clocksweep_scan_all_partitions=on
>
> (so it's another proof that code is fine, it's just algorithm that would
> have to adjusted)
>
> For benchmarks with pgbench -S for 100% local affinity vs 100% remote
> (I can do that with that pg_buffercache_set_partition() of mine), I'm
> getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
> I've spotted some another bug in from where we are fetching memory from
> unoptimal places if we are not on node#0, I'll need to dig into that more
> though. Another thing is that pgbench -S runs are much less demanding in
> terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
> seqconcurrscans.sql using the same amout of cores).
>
No opinion. I need to look at this closer.
>> The other thing I need to verify is how this behaves with
>> kernel.nr_hugepages. With some earlier versions it was easy to end in a
>> situation where everything seemed to work, but then much later the
>> kernel realized it does not have enough huge pages on a particular NUMA
>> node and crashed with a segfault (or was it sigbus?).
>
> It was SIGBUS and with this patchset I think we are fine: I have never
> witnessed this one crashing with SIGBUS.
>
Good. But I wonder if allocating just the precise number of huge pages
(per shared_memory_size_in_huge_pages) can prevent moving the partitions
to the correct node.
>> Of course, the other question is performance validation - does it even
>> help? I plan to repeat the various experiments mentioned in this thread
>> (by Andres and others) on available NUMA machines. But if someone has an
>> idea for another benchmark (and/or what metric to measure, not just the
>> usual duration), let me know.
>
> See above, but I think we would have to fix at at least: mbind() failure,
> and those VMAs disconnected regions.
>
Yes, the mbind() failure is a bug.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-17 12:13 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2026-06-17 12:13 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 16, 2026 at 2:39 PM Tomas Vondra <[email protected]> wrote:
>
>
>
> On 6/16/26 10:16, Jakub Wartak wrote:
> > On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <[email protected]> wrote:
> >>
> >> Hi,
> >
> > Hi Tomas, thanks for working on this.
> >
> >> Here's an updated version of the NUMA patch series, based on some recent
> >> discussions about this (some at pgconf.dev, but not only that),
> > [..]
> >
> > 1. 005 says:
> >
> > + * XXX We should enforce this in bufmgr.c, when initializing the partitions.
> > + */
> > +#define MAX_BUFFER_PARTITIONS 32
> >
> > but there isn't direct any check for checking if pg_numa_get_max_node() ->
> > numa_max_node() is not getting higher than allowed here. In theory this could
> > happen I think if ClockSweepPartitionIndex() would return
> > numa = numa_node_of_cpu()
> > on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
> > and that would cause accesing .balance[] without bounds.
> >
>
> Yes, this should be capped to the MAX_BUFFER_PARTITIONS.
>
> > 2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
> > this be padded/aligned somehow later in BufferStrategyControl which does
> > ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
> > to avoid contention/false sharing? (comments says it should be but it
> > doesn't seem so?), maybe the comment should be TODO for now? I have not
> > quantified any potential benefit
> >
> > With pahole after some hassle I've got:
> > struct ClockSweep {
[..]
> > pg_atomic_uint32 nextVictimBuffer; /* 16 4 */
[..]
> > /* size: 80, cachelines: 2, members: 11 */
> > /* sum members: 77, holes: 1, sum holes: 3 */
> > /* last cacheline: 16 bytes */
> > };
> > maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?
> >
>
> Possibly. Im not entirely happy with making the ClockSweep struct so
> much larger, but I haven't found a better way to store the counters
> needed for balancing. The only thing I can think of is storing it
> outside the struct, and maybe that's the right thing to do.
>
> But that assumes the current balancing approach is the right one.
Yeah, I'm just not sure if there is some wasted performance due to
false-sharing in very heavy benchmark scenarios (in theory the
nextVictimBuffer could bounce rather heavily).
> > 3. In 004 sched_getcpu() is used and mentioned how to check if it is available
> >
> > But my $0.02 (maybe not that important), but I've at least saw once where
> > (on EC2?) some clock_gettime() was very slow and that was because it was not
> > available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
> > always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
> > -> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
> > available, but slow and it would mean real syscall price (and that's not once
> > there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
> > mind, but I haven't checked further). The point is: wouldn't it be cheaper
> > that to be refreshed from time to time instead otherwise we risk some slow
> > code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
> > Or alternative is to have pg_test_numa proggie and this would be measuring
> > certain things about NUMA including timing of sched_getcpu (just like
> > pg_test_timing does for time), at least that could explain why somebody's
> > system/platform is slow.
> >
>
> Yes, I think we may need some sort of caching for this / check only
> sometimes. I haven't seen it to matter, but that may be luck and on
> other systems / platforms it may be worse.
Okay, so maybe an action point for us much later would be to try this on 2s+
ARM or some other much more rare setup just to see if we need to do it at all
(perhaps annotate it with TODO, I have short memory ;))
> > 4. Patch has problem (without fix for #8) that when number of available huge
> > pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
> > will use only first NUMA node. This might be a problem when starting mulitple
> > DBs (they will occupy first available NUMA):
> >
> > ### with s_b=8GB and nr_hugepages=1500 it's OK
> >
> > # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> > \; | grep 2048 | sort
> > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250
> >
> > ## note the correct split below for N0/N1..
> > # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> > 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> >
> > ### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
> > # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> > \; | grep 2048 | sort
> > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
> > ## all on N0...
> > # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> > 7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=4269 kernelpagesize_kB=2048
> >
> > I was even thinking go to lengths and add code for inspecting that /sys on
> > some later date that the kernel NUMA hugepages are really distributed
> > on the nodes as they should be (it's easy to end up on just 1 node out of
> > many; allocating via sysctl -w <higher> and then <lower> allocation is easy
> > way to force hugepages just to 1 node instead of many :o). I've hit the
> > problem multiple times, so we should bail out if we want NUMA and the
> > Buffer Blocks were just put on 1 node (instead of many).
> >
>
> How come the pg_numa_bind_to_node() calls don't move the parts to the
> correct node?
Well, if we were not checking error code mbind() it could behave in
non-deterministic way I think (you ask for MPOL_BIND, maybe it does something,
maybe it doesnt work but we continued anyway).
> If something is already using huge pages on the other nodes, then sure,
> it will fail. But I think that's OK - it's a best-effort thing. Maybe we
> should exit instead in this case?
Yes, I think we should exit with FATAL.
> > 5. In 005 we could mention more clealry what's the difference between
> > those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
> > in the defintion to make it easier to read, maybe copy-cat those earlier
> > descriptions there too as we already have:
> >
> > + * The balancing happens in intervals - it adjusts future allocations
> > + * based on stats about recent allocations, namely:
> > + *
> > + * - numBufferAllocs - number of allocations served by a partition
> > + *
> > + * - numRequestedAllocs - number of allocatios requested in a partition
> >
>
> I agree the explanation for this is not entirely clear.
>
> > 6. While at it, it would be helpful if we could reset the
> > pg_buffercache_partitions stats in some way (pg_buffercache_partitions
> > is very usefull).. or is there way to plug into main pg_reset functions?
> >
>
> Hmm, I was afraid it'd interfere with the balancing. I'm not sure it
> makes sense to reset just some of the fields - it'd make it much harder
> to interpret the counters. I'll think abou this.
Well, small hint: I find the the view very, very usefull in explaining
where/why memory is being allocated from. Maybe if we want to include it
in final version, then it might be worth (later on) to do it, if that
won't be included I'm fine just doing \watch 1 in psql to see what's
happening. Another hint is that maybe it doesnt belong to pg_buffercache
and would make it easier to integrate into pg_stat_reset*() -- dunno, how/
if extension can plug into it.
> > 7. If I add basic error checking for mbind() then it complains a lot, like
> > below with annotated strace -ffe mbind to show the point:
> >[..]
>
> I think this is a silly bug on my side, clearly the nodemask should have
> size > 0 even for node 0.
>
> > it doesn't report errors anymore and suprisngly hugepages in numa_maps are
> > altered from:
> > 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> >
> > to explicit "binds":
> > 7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
> > mapmax=6 N0=25 kernelpagesize_kB=2048
> > 7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=3 N0=7 kernelpagesize_kB=2048
> > 7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N1=7 kernelpagesize_kB=2048
> > 7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N2=7 kernelpagesize_kB=2048
> > 7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N3=7 kernelpagesize_kB=2048
> > 7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=3 N0=1 kernelpagesize_kB=2048
> > 7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
> > mapmax=2 N0=1023 kernelpagesize_kB=2048
> > 7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N0=1 kernelpagesize_kB=2048
> > 7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N1=1023 kernelpagesize_kB=2048
> > 7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N1=1 kernelpagesize_kB=2048
> > 7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N2=1023 kernelpagesize_kB=2048
> > 7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N3=1 kernelpagesize_kB=2048
> > 7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N3=1023 kernelpagesize_kB=2048
> > 7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
> > mapmax=6 N2=117 kernelpagesize_kB=2048
> >
> > so lots of VMAs were created (it could affect performance in some way, I think
> > for sure it would affect for worse fork() rates by postmaster for new conns).
> >
>
> Isn't this 4 VMAs for buffers ad 4 VMAs for buffer descriptors? Or maybe
> it's the parts that could not be mapped due to insufficient alignment?
Yes, like with patchset I'm getting WARNINGs:
WARNING: buffers for node 0 not well aligned
[0x7efc100f5000,0x7efc900f5000] aligned
[0x7efc10200000,0x7efc90000000]
WARNING: buffers descriptors for node 0 not well aligned
[0x7efc0c0f4f80,0x7efc0d0f4f41] aligned
[0x7efc0c200000,0x7efc0d000000]
WARNING: buffers for node 1 not well aligned
[0x7efc900f5000,0x7efd100f5000] aligned
[0x7efc90200000,0x7efd10000000]
WARNING: buffers descriptors for node 1 not well aligned
[0x7efc0d0f4f80,0x7efc0e0f4f41] aligned
[0x7efc0d200000,0x7efc0e000000]
WARNING: buffers for node 2 not well aligned
[0x7efd100f5000,0x7efd900f5000] aligned
[0x7efd10200000,0x7efd90000000]
WARNING: buffers descriptors for node 2 not well aligned
[0x7efc0e0f4f80,0x7efc0f0f4f41] aligned
[0x7efc0e200000,0x7efc0f000000]
WARNING: buffers for node 3 not well aligned
[0x7efd900f5000,0x7efe100f5000] aligned
[0x7efd90200000,0x7efe10000000]
WARNING: buffers descriptors for node 3 not well aligned
[0x7efc0f0f4f80,0x7efc100f4f41] aligned
[0x7efc0f200000,0x7efc10000000]
LOG: starting PostgreSQL 19beta1 on x86_64-linux, compiled by
gcc-12.2.0, 64-bit
relevant numa_maps for postmaster:
7efc0c200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N0=7 kernelpagesize_kB=2048
7efc0d000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0d200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N1=7 kernelpagesize_kB=2048
7efc0e000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0e200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N2=7 kernelpagesize_kB=2048
7efc0f000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0f200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N3=7 kernelpagesize_kB=2048
7efc10000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=3 N3=1 kernelpagesize_kB=2048
7efc10200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
N0=1023 kernelpagesize_kB=2048
7efc90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048
7efc90200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
N1=1023 kernelpagesize_kB=2048
7efd10000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N0=1 kernelpagesize_kB=2048
7efd10200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
N2=1023 kernelpagesize_kB=2048
7efd90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N2=1 kernelpagesize_kB=2048
7efd90200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
N3=1023 kernelpagesize_kB=2048
7efe10000000 default file=/anon_hugepage\040(deleted) huge dirty=116
mapmax=6 N1=116 kernelpagesize_kB=2048
so if You take just 7efc10200000..7efc90000000 (from buffers@node0, 1st line)
and zoom in/grep You'll get:
7efc10200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
N0=1023 kernelpagesize_kB=2048
7efc90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048
so @ 7efc90000000 it's Node3=1 hugepage (so it's wrong by 1x 2048kb offset)
so it contains the tail of node 0 buffers and the head of node - and is
excluded from both mbind() calls. At first I've spotted this in 003/
BufferPartitionsInit() with:
cstartptr = (char *) &BufferDescriptors[buff_first];
endptr = (char *) &BufferDescriptors[buff_last] + 1; // <-- BUG?
with
endptr = (char *) &BufferDescriptors[buff_last + 1];
but got the same issue, so I've forced the mbind() to use 2x TYPEALIGN_DOWN
to cover with policy everything and got:
7fc134e00000 default file=/anon_hugepage\040(deleted) huge dirty=24
mapmax=6 N3=24 kernelpagesize_kB=2048
7fc137e00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=4 N0=8 kernelpagesize_kB=2048
7fc138e00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=2 N1=8 kernelpagesize_kB=2048
7fc139e00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=3 N2=8 kernelpagesize_kB=2048
7fc13ae00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=2 N3=8 kernelpagesize_kB=2048
7fc13be00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1024
N0=1024 kernelpagesize_kB=2048
7fc1bbe00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1024
N1=1024 kernelpagesize_kB=2048
7fc23be00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1024
mapmax=2 N2=1024 kernelpagesize_kB=2048
7fc2bbe00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1024
N3=1024 kernelpagesize_kB=2048
7fc33be00000 default file=/anon_hugepage\040(deleted) huge dirty=116
mapmax=6 N1=116 kernelpagesize_kB=2048
that was with:
@@ -351,7 +351,7 @@ BufferPartitionsInit(void)
}
/* best effort: align the pointers, so that
the mbind() works */
- startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ startptr = (char *)
TYPEALIGN_DOWN(numa_page_size, startptr);
endptr = (char *)
TYPEALIGN_DOWN(numa_page_size, endptr);
@@ -374,7 +374,7 @@ BufferPartitionsInit(void)
}
/* best effort: align the pointers, so that
the mbind() works */
- startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ startptr = (char *)
TYPEALIGN_DOWN(numa_page_size, startptr);
endptr = (char *)
TYPEALIGN_DOWN(numa_page_size, endptr);
Looks way better, but it was wild guess (TBH I think I've prefered the
previous patchset version where the input shm addresses were already aligned,
but if You and others say it's easier route then sure)
> > To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
> > single HP page being somehow left/missed in address calculations, but I
> > haven't pressed this harder.
> >
> > NOTE: the patch works even without this fix, but I believe if got non-0 we
> > cannot reliably trust the optimizer memory layout has been deployed (I suspect
> > it's some kind luck it sharded the shm based on number of hugepages available)
> >
>
> I'm not sure I understand what you mean. What non-0?
Non-0 return code from mbind() as the mbind() failed with -1, but it did somehow
alter numa pages probably(?), but without creating specific "isolated" VMA
explicit "bind" policy, probably maybe it did not even work at all..
> >
> >> questions
> >> ---------
> >>
> >> At this point, my main question is whether there's a better way to
> >> partition clock-sweep and/or do the balancing of allocations between
> >> partitions. I believe it does work, but I have a feeling there might be
> >> a more elegant way to do this kind of stuff (like an established
> >> balancing algorithm of some sort).
> >
> >
> > 8. The crux of this email and stuff I wanted to further discuss, when server
> > is started with on this 4-NUMA box with
> > * numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
> > on node#0
> > * the shm split onto 4 nodes properly
> > * s_b still just 8GB (with ideal split),
> > * DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
> > * pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
> > \set num (:client_id % 8) + 1
> > select sum(octet_length(filler)) from pgbench_accounts_:num;
> > * mpstat repors correctly just node#0 used
> >
> > a. with the patch for GUCs with numa on and defaults two clocksweep settings
> > on, I'm getting:
> >
> > latency average = 3252.254 ms
> > latency stddev = 72.011 ms
> >
> > b. with debug_clocksweep_balance=off, I'm realiably getting
> >
> > latency average = 2688.742 ms
> > latency stddev = 61.738 ms
> >
> > so IMHO clocksweep partitioning is cool, but if we are discussing the current
> > balancing logic leaves some juice on the table from the most optimized variant
> > (~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
> > was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).
> >
>
> How does this compare to master, i.e. without these NUMA patches?
I had some problems comparing, but with "perfect setup" that includes the
following:
- pgbench/clients on node#0
- backends running on node#0
- hugepages memory on node#0..3, but with with this patchset and those goodies:
debug_clocksweep_balance=on
debug_clocksweep_balance_recalc=off
debug_clocksweep_scan_all_partitions=on
(so technically node0 backends accessing just Buffers / Buffers Desc from
node0, technically node0 weights: "{100,0,0,0}")
- with today's new discovery for me that Linux's kernel VFS cache is also
having also first-touch (!) NUMA policy and really important here (so
VFS cached data also alters results of testing wildly!), so I had to
force unloading VFS cache and force-loading it into node#0, I've was
getting for seqconcurrscans:
latency average = 2701.705 ms
latency stddev = 111.608 ms
vs master, huh, but which scenario? the default one without any affinity?
- assuming you get the split shm split like "N0=1059 N1=1299 N2=879 N3=1031"
but that's appeneded-only (so not interleaved (?)) and you even risk
having shm placed on just __one__ node (if it is big enough and free
enough)
- if you go straight to benchmarking it (CPU hits random nodes)
latency average = 3439.200 ms
latency stddev = 580.501 ms
- with backends forked() to node#0 (numactl --cpunodebind=0 pg_ctl start)
latency average = 4937.543 ms
latency stddev = 573.841 ms
because of random VFS cache placement (I imagine it as flow of on node0 CPUs
a. nextVictimBuffer contention
b. getBuffer() - fetch random shm memory from random NUMA node
c. pread() - fetch from VFS cache but from *remote* NUMA node
)
- with backends forked() to node#0 and pinned VFS cached fully on node#0 too
latency average = 4518.651 ms
latency stddev = 797.369 ms
(but this is still Buffers from other nodes)
- same as above above and numactl interleaved shm:
latency average = 3792.016 ms
latency stddev = 825.186 ms
- same as above and interleaved shm, but without pining to CPUs on specifc
node and ensure random VFS cache vs nodes:
latency average = 2913.813 ms
latency stddev = 352.552 ms
- but the moment you read anything reads base/ into VFS cache to particular
node (imagine pg_prewarm or even just tar) assuming it was not there it
also pins that to that node memory and you'll get:
latency average = 3594.180 ms
latency stddev = 851.949 ms
> > Dunno if it should be optimized further, certainly we'll get reports from
> > quick benchmarks run by people that PG 20 could be *slower* because.. well,
> > they got (sub)optimal layout during startup (all HP on 1 node and some
> > query hitting just that one query with local affinity and this is visible
> > to naked eye). I was re-reading thread and Andres also wrote "We should use
> > the partitioned clock sweep to default to using local memory as long as
> > possible."
> >
> > So two further ideas:
> >
> > I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
> > itself, couldn't we? If we are using capped ring buffer, we could indicate
> > that we want it just from local node as priority disregarding weights (?).
> > Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
> > there would be some potential issue with sync-scan-table code though.
> > With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
> > prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
> > for ideal distribution amongst all NUMA nodes.
> >
>
> Yes, I think it might make sense to disable balancing in these cases.
OK, I did not code anything of that as
> > II. what if we could track if the relation is just all-local-access?
> >
> > Another idea is that if we would know that's it's just us working on some
> > relation (created by us; or it's not being touched remotley) then we could
> > ask for local-only memory affinity. So something like this:
> >
> > a. in case of locally-only access rels =>
> > ask for local memory first
> > if that fails failback to weighted RR (so to to weights)
> > b. in case other rels => weighted RR (so to to weights) directly
> >
> > The tracking of the fact that Buffer was accessed just locally or remotley
> > itself is not hard to imagine (by using some free "bits" in BufferDesc.
> > "state" where refcount/usagecount itself are stored, well at least 4 bits
> > for my 4 nodes, but there's plenty of left there), I have some PoC for
> > that but that's just per-Buffer tracking of "was this Buffer accessed by
> > remote nodes", but I'm completley lost how to make transition to the
> > is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
> > optimization (we would need some shared infra just for tracking such info;
> > assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
> > 2..4 bits, that's already huge number: we are talking GBs of shm mem).
> >> BTW: I've been experimenting with this patchset and added couple of things
> > (see attached), and with I'm able to get optimal just by forcing affinity
> > too using that earlier bench:
> >
> > latency average = 2512.929 ms
> > latency stddev = 97.525 ms
> >
> > and that was with pure 100% affinity to my local node:
> >
> > select pg_buffercache_set_partition(0, '{100,0,0,0}');
> > debug_clocksweep_balance_recalc=off
> > debug_clocksweep_balance=on
> > debug_clocksweep_scan_all_partitions=on
> >
> > (so it's another proof that code is fine, it's just algorithm that would
> > have to adjusted)
> >
> > For benchmarks with pgbench -S for 100% local affinity vs 100% remote
> > (I can do that with that pg_buffercache_set_partition() of mine), I'm
> > getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
> > I've spotted some another bug in from where we are fetching memory from
> > unoptimal places if we are not on node#0, I'll need to dig into that more
> > though. Another thing is that pgbench -S runs are much less demanding in
> > terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
> > seqconcurrscans.sql using the same amout of cores).
> >
>
> No opinion. I need to look at this closer.
Great !
> >> The other thing I need to verify is how this behaves with
> >> kernel.nr_hugepages. With some earlier versions it was easy to end in a
> >> situation where everything seemed to work, but then much later the
> >> kernel realized it does not have enough huge pages on a particular NUMA
> >> node and crashed with a segfault (or was it sigbus?).
> >
> > It was SIGBUS and with this patchset I think we are fine: I have never
> > witnessed this one crashing with SIGBUS.
> >
>
> Good. But I wonder if allocating just the precise number of huge pages
> (per shared_memory_size_in_huge_pages) can prevent moving the partitions
> to the correct node.
>
Not sure I understand (?) how's that related to SIGBUS?
-J.
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-24 20:26 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2026-06-24 20:26 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Here's an updated patch series, with only minor changes to fix the mbind
issues:
1) It uses the correct nodemask size, so that the mbind actually binds
the partition to the right node.
2) It aligns the start/end pointers so that there no pages are left with
the default memory policy. So now there should be only the bind:N
entries, not the single-page "default" ones. This means the last page of
a partition can be mapped to a different node, but that seems fine (in
the end it could have happened with the old approach too).
I've also included Jakub's "goodies" patch with the additional GUCs.
Those seem potentially useful to development.
I have some results from a new round of benchmarks, and it's a bit
disappointing. Or rather, there seem to be some issues that I can't
figure out, causing regressions.
Consider a very simple test, doing a lot of sequential scans to put a
fair amount of pressure on the clocksweep / buffer replacement. There's
a .tgz with the benchmark script attached, but it does about this:
* Initialize a pgbench database with scale 2000 (so ~30GB, about twice
the shared buffers).
* Uses --partitions=100, so that the partitions are small enough not to
trigger the 1/4 threshold (i.e. not use circular buffers).
* Does runs with custom script, forcing sequential scans of the table,
with two queries:
select count(1) from pgbench_accounts;
select * from pgbench_accounts offset 1000000000;
Those are called "count" and "offset" in the results. The script forces
serial sequential scans (no index scans, no parallelism), and does runs
with 1, 8 and 32 clients (this is an old-ish xeon with 44 physical cores
on two sockets, 2 NUMA nodes).
I did runs with "master" and the all the 7 patches, with the NUMA stuff
enabled/disabled since 0003 (which adds it). See the two PDFs with more
complete results, but here's the "count" query for a subset of the
patches (the omitted ones behave similarly to what's shown here).
This chart is for median latency (in milliseconds):
clients master 0003 0004 0003/on 0004/on
-------------------------------------------------------------
1 12767 12582 14509 12807 15307
8 14383 14355 14149 14069 16165
32 14756 15198 14836 14984 17128
--------------------------------------------------------
1 103% 114% 100% 120%
8 101% 98% 98% 112%
32 102% 101% 102% 116%
The percentages are compared to "master", the columns with "/on" are
with shared_buffers_numa=on.
Clearly, there's no chance with 0003 (which binds shared buffer
partitions to NUMA nodes, even if that's enabled). The differences are
within noise, pretty much, for all client counts.
Then 0004 gets applied, which partitions the clock sweep. And well, that
doesn't go particularly well. There is a bit of a regression even with
numa=off, but it kinda recovers with the following patches. But with
numa=on, there's a consistent ~10% regression (give or take).
I've spent a fair bit of time investigating what's causing this, but so
far I have nothing. I assume it's something silly in the patches
partitioning the clocksweep, or maybe the approach is flawed in some
way. Not sure :-(
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20260624-0001-Add-shmem_populate-and-shmem_interleave-GU.patch (4.9K, ../../[email protected]/2-v20260624-0001-Add-shmem_populate-and-shmem_interleave-GU.patch)
download | inline diff:
From e6d192ae573d63b5cb5c71773047b97c97eb7961 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 3 Jun 2026 16:30:28 +0200
Subject: [PATCH v20260624 1/7] Add shmem_populate and shmem_interleave GUCs
- shmem_populate - Forces mmap() with MAP_POPULATE, which faults all
memory pages backing the shared memory segment.
- shmem_interleave - Applies NUMA interleaving on the whole shared
memory segment, to balance allocations between nodes.
---
src/backend/port/sysv_shmem.c | 47 +++++++++++++++++++++++
src/backend/utils/misc/guc_parameters.dat | 14 +++++++
src/include/miscadmin.h | 4 ++
3 files changed, 65 insertions(+)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 2e3886cf9fe..9eaff838a04 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -27,6 +27,10 @@
#include <sys/shm.h>
#include <sys/stat.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "portability/mem.h"
@@ -98,6 +102,10 @@ void *UsedShmemSegAddr = NULL;
static Size AnonymousShmemSize;
static void *AnonymousShmem = NULL;
+/* GUCs */
+bool shmem_populate = false; /* MAP_POPULATE */
+bool shmem_interleave = false; /* NUMA interleaving */
+
static void *InternalIpcMemoryCreate(IpcMemoryKey memKey, Size size);
static void IpcMemoryDetach(int status, Datum shmaddr);
static void IpcMemoryDelete(int status, Datum shmId);
@@ -604,6 +612,21 @@ CreateAnonymousSegment(Size *size)
int mmap_errno = 0;
int mmap_flags = MAP_SHARED | MAP_ANONYMOUS | MAP_HASSEMAPHORE;
+ /* If requested, populate the shared memory by MAP_POPULATE. */
+ if (shmem_populate)
+ mmap_flags |= MAP_POPULATE;
+
+#ifdef USE_LIBNUMA
+ /*
+ * If requested, interleave the shared memory by setting a memory policy
+ * before the mmap() call. This really matters only with MAP_POPULATE,
+ * because without page faults the memory does not actually get placed
+ * to the nodes. But without MAP_POPULATE it's virtually free.
+ */
+ if (shmem_interleave)
+ numa_set_interleave_mask(numa_all_nodes_ptr);
+#endif
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -665,6 +688,30 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+#ifdef USE_LIBNUMA
+ /*
+ * If set the policy to interleaving by numa_set_membind(), undo it now by
+ * setting the policy to localalloc. With MAP_POPULATE, all the pages were
+ * faulted and are now interleaved on the available nodes.
+ *
+ * To handle the case without MAP_POPULATE, apply the interleaving policy to
+ * the shared memory segment allocated by mmap() before touching it in any
+ * way, so that it gets placed on the correct node on first access.
+ *
+ * This matters especially with huge pages, where it's possible to run out
+ * of huge pages on some nodes and then crash. By explicitly interleaving
+ * the whole segment, that's much less likely.
+ */
+ if (shmem_interleave)
+ {
+ /* undo the policy set by numa_set_membind() earlier */
+ numa_set_localalloc();
+
+ /* set interleaving policy for not yet faulted memory */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+ }
+#endif
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index afaa058b046..f15e74198c5 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -740,6 +740,20 @@
ifdef => 'DEBUG_NODE_TESTS_ENABLED',
},
+{ name => 'debug_shmem_interleave', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Forces interleaving for the whole shared memory segment.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_interleave',
+ boot_val => 'false'
+},
+
+{ name => 'debug_shmem_populate', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Populates (faults) the whole shared memory segment using MAP_POPULATE.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_populate',
+ boot_val => 'false'
+},
+
{ name => 'debug_write_read_parse_plan_trees', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Set this to force all parse and plan trees to be passed through outfuncs.c/readfuncs.c, to facilitate catching errors and omissions in those modules.',
flags => 'GUC_NOT_IN_SAMPLE',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 7170a4bff98..bf38aa6baa2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -213,6 +213,10 @@ extern PGDLLIMPORT Oid MyDatabaseTableSpace;
extern PGDLLIMPORT bool MyDatabaseHasLoginEventTriggers;
+extern PGDLLIMPORT bool shmem_populate;
+extern PGDLLIMPORT bool shmem_interleave;
+
+
/*
* Date/Time Configuration
*
--
2.54.0
[text/x-patch] v20260624-0002-Infrastructure-for-partitioning-of-shared-.patch (14.3K, ../../[email protected]/3-v20260624-0002-Infrastructure-for-partitioning-of-shared-.patch)
download | inline diff:
From 1f3fe6ec274f459ca4de440cf9da4eb43818e48c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:09:33 +0200
Subject: [PATCH v20260624 2/7] Infrastructure for partitioning of shared
buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep or to make the
partitioning NUMA-aware).
The registry is a small array of BufferPartition entries in shared
memory, with partitions sized to be a fair share of shared buffers.
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
* The buffers are divided as evenly as possible, with the first couple
partitions possibly getting an extra buffer.
---
contrib/pg_buffercache/Makefile | 3 +-
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 142 ++++++++++++++++++
src/include/storage/buf_internals.h | 5 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 0e618f66aec..7fd5cdfc43d 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,8 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql \
+ pg_buffercache--1.7--1.8.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
new file mode 100644
index 00000000000..d62b8339bfc
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -0,0 +1,23 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index 11499550945..d2fa8ba53ba 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'
+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 510455998aa..6c9838b4efc 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,6 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -77,6 +78,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -922,3 +924,87 @@ pg_buffercache_mark_dirty_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of shared buffers.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 1407c930c56..e593b02e0ca 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -26,10 +26,12 @@ char *BufferBlocks;
ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsRegistry;
static void BufferManagerShmemRequest(void *arg);
static void BufferManagerShmemInit(void *arg);
static void BufferManagerShmemAttach(void *arg);
+static void BufferPartitionsInit(void);
const ShmemCallbacks BufferManagerShmemCallbacks = {
.request_fn = BufferManagerShmemRequest,
@@ -69,6 +71,9 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
+/* number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
/*
* Register shared memory area for the buffer pool.
@@ -97,6 +102,13 @@ BufferManagerShmemRequest(void *arg)
.ptr = (void **) &BufferIOCVArray,
);
+ ShmemRequestStruct(.name = "Buffer Partition Registry",
+ .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ /* Align descriptors to a cacheline boundary. */
+ .alignment = PG_CACHE_LINE_SIZE,
+ .ptr = (void **) &BufferPartitionsRegistry,
+ );
+
/*
* The array used to sort to-be-checkpointed buffer ids is located in
* shared memory, to avoid having to allocate significant amounts of
@@ -119,6 +131,12 @@ BufferManagerShmemRequest(void *arg)
static void
BufferManagerShmemInit(void *arg)
{
+ /*
+ * Initialize the buffer partition registry first, before other parts
+ * have a chance to touch the memory.
+ */
+ BufferPartitionsInit();
+
/*
* Initialize all the buffer headers.
*/
@@ -151,3 +169,127 @@ BufferManagerShmemAttach(void *arg)
WritebackContextInit(&BackendWritebackContext,
&backend_flush_after);
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsRegistry->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsRegistry->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsRegistry->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsRegistry->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * BufferPartitionsInit
+ * Initialize registry of buffer partitions.
+ */
+static void
+BufferPartitionsInit(void)
+{
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
+ int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+
+ int num_buffers = part_buffers;
+ if (n < remaining_buffers)
+ num_buffers += 1;
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+/*
+ * BufferPartitionCount
+ * Returns the number of partitions created.
+ */
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsRegistry->npartitions;
+}
+
+/*
+ * BufferPartitionGet
+ * Returns information about a partition at the provided index.
+ *
+ * The returned information is first/last buffer, number of buffers.
+ */
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 89615a254a3..e5a887b9969 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -411,9 +411,14 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsRegistry;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 6837b35fc6d..79a3f44747a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -155,6 +155,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c5db6ca6705..45fc91fcb97 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -365,6 +365,8 @@ BufferHeapTupleTableSlot
BufferLockMode
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.54.0
[text/x-patch] v20260624-0003-NUMA-shared-buffers-partitioning.patch (26.8K, ../../[email protected]/4-v20260624-0003-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 5a8f757fea9a2755d1c4fadad067352f4ab307be Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 23:27:13 +0200
Subject: [PATCH v20260624 3/7] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc.
In cases like pre-warming a database from a single worker (e.g. using
pg_prewarm), we may end up with severely unbalanced memory distribution
(with most memory located on a single NUMA node). Unbalanced allocation
may put a lot of pressure on the memory system on a small number of NUMA
nodes, limiting the bandwidth etc.
With zone_reclaim, the kernel would eventually move some of the memory
to other nodes, but that tends to take a long time and is unpredictable.
This change forces even distribution of shared buffers on all NUMA
nodes, improving predictability, reducing the time needed for warmup
during benchmarking, etc. It's also less dependent on what the CPU
scheduler decides to do (which cores get used for the warmup.)
The effect is similar to
numactl --interleave=all
in that the buffers are distributed on the NUMA nodes evenly, but
there's also a number of important differences.
Firstly, it's applied only to shared buffers (and buffer descriptors),
not to the whole shared memory segment. It's possible to enable memory
interleaving using the shmem_interleave GUC, introduced in an earlier
patch in this series.
NUMA works at the granularity of a memory page, which is typically
either 4K or 2MB (hugepage), but other sizes are possible. For systems
where NUMA matters, we expect large amounts of memory (hundreds of
gigabytes) and hugepages enabled. But not necessarily.
The partitioning scheme is best-effort with respect to memory page size.
The shared buffers do not "align" with memory pages (i.e. a partition
may not end at the memory page boundary), in which case we simply locate
just the section of the partition with complete memory pages. This means
there may be ~one unmapped memory page between partitions. Considering
the expected amounts of memory, this is negligible, and the alternative
would be a significant amount of complexity to align the pages and
enforce "allowed" partition sizes.
Buffer descriptors are affected by this too, and the effect may be more
significant, simply because the descriptors are much smaller (~64B). So
the array is smaller, and a single 2MB memory page is worth ~32K buffer
descriptors. But with large systems it's still negligible.
The "buffer partitions" may not be 1:1 with NUMA nodes. We want to allow
clock-sweep partitioning even on non-NUMA systems, or when running only
on a small number of NUMA nodes. There's a minimal number of partitions
(default: 4), and a node may get multiple partitions. Nodes always get
the same number of partitions (e.g. with 3 NUMA nodes there will be 6
partitions in total, as each node gets 2 partitions).
The feature is enabled by dshared_buffers_numa GUC (default: false).
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/buf_init.c | 251 ++++++++++++++++--
src/backend/storage/buffer/freelist.c | 9 +
src/backend/utils/misc/guc_parameters.dat | 6 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/port/pg_numa.h | 7 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 8 +
src/port/pg_numa.c | 112 ++++++++
10 files changed, 399 insertions(+), 36 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index d62b8339bfc..a6e49fd1652 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer); -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 6c9838b4efc..46b6b85a2e3 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -955,11 +955,13 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -977,28 +979,32 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index e593b02e0ca..4ed79db7b49 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,12 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proclist.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -71,9 +79,12 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
-/* number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+bool shared_buffers_numa = false;
/*
* Register shared memory area for the buffer pool.
@@ -81,6 +92,10 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
static void
BufferManagerShmemRequest(void *arg)
{
+ int nparts;
+
+ BufferPartitionsCalculate(NULL, &nparts, NULL);
+
ShmemRequestStruct(.name = "Buffer Descriptors",
.size = NBuffers * sizeof(BufferDescPadded),
/* Align descriptors to a cacheline boundary. */
@@ -103,7 +118,7 @@ BufferManagerShmemRequest(void *arg)
);
ShmemRequestStruct(.name = "Buffer Partition Registry",
- .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ .size = nparts * sizeof(BufferPartition),
/* Align descriptors to a cacheline boundary. */
.alignment = PG_CACHE_LINE_SIZE,
.ptr = (void **) &BufferPartitionsRegistry,
@@ -134,6 +149,10 @@ BufferManagerShmemInit(void *arg)
/*
* Initialize the buffer partition registry first, before other parts
* have a chance to touch the memory.
+ *
+ * Also moves memory to different NUMA nodes (if enabled by a GUC).
+ * Do this before the loop that initializes buffer headers etc. which
+ * may fault some of the memory pages etc.
*/
BufferPartitionsInit();
@@ -231,35 +250,210 @@ BufferPartitionsInit(void)
{
int buffer = 0;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
- int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+ int nnodes,
+ npartitions,
+ npartitions_per_node;
- BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ int buffers_per_partition,
+ buffers_remaining;
- for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
- {
- BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+ /* calculate partitioning parameters */
+ BufferPartitionsCalculate(&nnodes, &npartitions, &npartitions_per_node);
+
+ /* paranoia */
+ Assert(nnodes > 0);
+ Assert(npartitions >= MIN_BUFFER_PARTITIONS);
+ Assert((npartitions % nnodes) == 0);
+ Assert((npartitions_per_node * nnodes) == npartitions);
- int num_buffers = part_buffers;
- if (n < remaining_buffers)
- num_buffers += 1;
+ BufferPartitionsRegistry->nnodes = nnodes;
+ BufferPartitionsRegistry->npartitions = npartitions;
+ BufferPartitionsRegistry->npartitions_per_node = npartitions_per_node;
- remaining_buffers -= num_buffers;
+ /* regular partition size, the first couple get an extra buffer */
+ buffers_per_partition = (NBuffers / npartitions);
+ buffers_remaining = (NBuffers % buffers_per_partition);
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ /* should have all the buffers */
+ Assert((buffers_per_partition * npartitions + buffers_remaining) == NBuffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ /*
+ * Now walk the partitions, and set the buffer range. Optionally, place
+ * the partitions on a given node (for all partitions at once).
+ */
+ for (int n = 0; n < nnodes; n++)
+ {
+ for (int p = 0; p < npartitions_per_node; p++)
+ {
+ int idx = (n * npartitions_per_node) + p;
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ /*
+ * Assign to the NUMA node, but only with shared_buffers_numa=on.
+ *
+ * XXX we should get an actual node ID from the mask, in case the
+ * task is restricted to only some nodes.
+ */
+ part->numa_node = (shared_buffers_numa) ? n : -1;
+
+ /* The first couple partitions may get an extra buffer. */
+ part->num_buffers = buffers_per_partition;
+ if (idx < buffers_remaining)
+ part->num_buffers += 1;
+
+ /* remember the buffer range */
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (part->num_buffers - 1);
+
+ /* remember start of the next partition */
+ buffer += part->num_buffers;
+ }
- buffer += num_buffers;
+#ifdef USE_LIBNUMA
+ /*
+ * Now try to locate buffers and buffer descriptors to the node (all
+ * partitions for the node at once).
+ */
+ if (shared_buffers_numa)
+ {
+ Size numa_page_size = pg_numa_page_size();
+
+ int part_first,
+ part_last,
+ buff_first,
+ buff_last;
+
+ char *startptr,
+ *endptr;
+
+ /* first/last partition for this node */
+ part_first = (n * npartitions_per_node);
+ part_last = part_first + (npartitions_per_node - 1);
+
+ /* buffers (blocks) */
+
+ /* first/last buffer */
+ buff_first = BufferPartitionsRegistry->partitions[part_first].first_buffer;
+ buff_last = BufferPartitionsRegistry->partitions[part_last].last_buffer;
+
+ /* beginning of the first block, end of last block */
+ startptr = BufferBlocks + ((Size) buff_first * BLCKSZ);
+ endptr = BufferBlocks + ((Size) (buff_last + 1) * BLCKSZ);
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers for node %d not well aligned [%p,%p]",
+ n, startptr, endptr);
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN_DOWN(numa_page_size, startptr);
+
+ /* the last partition aligns to the end of the buffer */
+ if (n == (nnodes - 1))
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+ else
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ if (pg_numa_bind_to_node(startptr, endptr, n) != 0)
+ elog(WARNING, "failed to bind shared buffers partition to node %d", n);
+
+ /* buffer descriptors */
+
+ /* beginning of the first descriptor, end of last descriptor */
+ startptr = (char *) &BufferDescriptors[buff_first];
+ endptr = (char *) &BufferDescriptors[buff_last] + 1;
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers descriptors for node %d not well aligned [%p,%p]",
+ n, startptr, endptr);
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN_DOWN(numa_page_size, startptr);
+
+ if (n == (nnodes - 1))
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+ else
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ if (pg_numa_bind_to_node(startptr, endptr, n) != 0)
+ elog(WARNING, "failed to bind shared buffer descriptors partition to node %d", n);
+ }
+#endif
}
AssertCheckBufferPartitions();
}
+/*
+ * BufferPartitionsCalculate
+ * Pick number of buffer partitions for the number of nodes and
+ * MIN_BUFFER_PARTITIONS.
+ *
+ * Picks the smallest number of partitions higher thah MIN_BUFFER_PARTITIONS,
+ * such that all nodes have the same number of partitions.
+ *
+ * This is best-effort with respect to size of the partitions. It's possible
+ * the partitions are not a perfect multiple of page size, in which case
+ * we set location only for the part where that is possible. The buffers on
+ * the "boundary" may get located up on arbitrary nodes.
+ *
+ * The extra complexity of figuring out the right "partition size" is not
+ * worth it, and it can lead to some partitions being much smaller. This way
+ * we end up with partitions of almost exactly the same size (one BLCKSZ is
+ * the largest difference).
+ *
+ * We expect shared buffers to be much larger than page size (at least on
+ * system where NUMA is a relevant feature), so the number of "not located"
+ * buffers should be a negligible fraction. This only affects pages between
+ * partitions for different nodes, so (nodes-1) pages. This is certainly
+ * fine with 2MB huge pages, but even with 1GB pages it should be OK (as
+ * such systems should have humongous amounts of memory).
+ *
+ * It also means we don't need to worry about memory page size before knowing
+ * if huge pages got used (which we only learn during allocation).
+ */
+void
+BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ int nnodes,
+ nparts,
+ nparts_per_node;
+
+#if USE_LIBNUMA
+ nnodes = numa_num_configured_nodes();
+ nparts_per_node = 1; /* at least one partition per node */
+
+ while ((nparts_per_node * nnodes) < MIN_BUFFER_PARTITIONS)
+ nparts_per_node++;
+
+ nparts = (nnodes * nparts_per_node);
+#else
+ /* without NUMA, assume there's just one node */
+ nnodes = 1;
+ nparts = MIN_BUFFER_PARTITIONS;
+ nparts_per_node = MIN_BUFFER_PARTITIONS;
+#endif
+
+ if (num_nodes)
+ *num_nodes = nnodes;
+
+ if (num_partitions)
+ *num_partitions = nparts;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = nparts_per_node;
+}
+
/*
* BufferPartitionCount
* Returns the number of partitions created.
@@ -277,13 +471,14 @@ BufferPartitionCount(void)
* The returned information is first/last buffer, number of buffers.
*/
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
{
BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -293,3 +488,17 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+void
+BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ if (num_nodes)
+ *num_nodes = BufferPartitionsRegistry->nnodes;
+
+ if (num_partitions)
+ *num_partitions = BufferPartitionsRegistry->npartitions;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = BufferPartitionsRegistry->npartitions_per_node;
+}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index fdb5bad7910..53ef5239e8d 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,15 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f15e74198c5..2e71c04282c 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2724,6 +2724,12 @@
max => 'INT_MAX / 2',
},
+{ name => 'shared_buffers_numa', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
+ short_desc => 'Locate partitions of shared buffers (and descriptors) to NUMA nodes.',
+ variable => 'shared_buffers_numa',
+ boot_val => 'false',
+},
+
{ name => 'shared_memory_size', type => 'int', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows the size of the server\'s main shared memory area (rounded up to the nearest MB).',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_UNIT_MB | GUC_RUNTIME_COMPUTED',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..c0f79c779cc 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -142,6 +142,7 @@
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
# (change requires restart)
+#shared_buffers_numa = off # NUMA-aware partitioning
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB # min 64kB
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 1b668fe1d91..8fe4d4ab7e3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,13 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+extern PGDLLIMPORT int pg_numa_bind_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e5a887b9969..e944cee2e91 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -365,10 +365,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -378,7 +378,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -416,8 +416,12 @@ extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
extern int BufferPartitionCount(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
+extern void BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 79a3f44747a..1cf09e8fb7c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -158,10 +158,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -170,7 +172,9 @@ typedef struct BufferPartition
/* an array of information about all partitions */
typedef struct BufferPartitions
{
+ int nnodes; /* number of NUMA nodes */
int npartitions; /* number of partitions */
+ int npartitions_per_node; /* for convenience */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -206,6 +210,7 @@ extern PGDLLIMPORT const PgAioHandleCallbacks aio_local_buffer_readv_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
+extern PGDLLIMPORT bool shared_buffers_numa;
/* in localbuf.c */
extern PGDLLIMPORT int NLocBuffer;
@@ -390,6 +395,9 @@ extern void MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied,
int32 *buffers_already_dirty,
int32 *buffers_skipped);
+/* in buf_init.c */
+extern int BufferGetNode(Buffer buffer);
+
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 8954669273a..66985f32db3 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -118,6 +121,94 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ int ret;
+ struct bitmask *nodemask;
+
+ if (node < 0)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ nodemask = numa_allocate_nodemask();
+ if (nodemask == NULL)
+ {
+ errno = ENOMEM;
+ return -1;
+ }
+
+ numa_bitmask_setbit(nodemask, node);
+
+ /*
+ * MPOL_BIND places the pages strictly on the node, and MPOL_MF_MOVE migrates
+ * pages already faulted in to that node. If mbind() fails, leave the default
+ * placement in effect, and report the failure.
+ */
+ ret = mbind(startptr, (endptr - startptr),
+ MPOL_BIND, nodemask->maskp, nodemask->size, MPOL_MF_MOVE);
+
+ numa_free_nodemask(nodemask);
+
+ return ret;
+}
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
#else
/* Empty wrappers */
@@ -140,4 +231,25 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+Size
+pg_numa_page_size(void)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
--
2.54.0
[text/x-patch] v20260624-0004-clock-sweep-basic-partitioning.patch (34.0K, ../../[email protected]/5-v20260624-0004-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 8265b556de61802983a3fefc31e0e2e071b456ed Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:16:55 +0200
Subject: [PATCH v20260624 4/7] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 202 +++++++----
src/backend/storage/buffer/freelist.c | 333 ++++++++++++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 486 insertions(+), 104 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index a6e49fd1652..92176fed7f8 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -14,7 +14,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 46b6b85a2e3..b07fafda0d9 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -963,6 +963,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -984,12 +992,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -1005,6 +1023,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d6c0cc1f6d4..7bedaf8987a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3826,33 +3826,34 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
- *
- * This is called periodically by the background writer process.
+ * Information saved between calls so we can determine the strategy point's
+ * advance rate and avoid scanning already-cleaned buffers.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX Does it actually make sense to split all of this information per
+ * partition? For example, does per-partition advance rate mean anything?
+ * Maybe we should have a global advance rate? Although, if we want to
+ * keep enough clean buffers in each partition, maybe having per-partition
+ * rates makes sense.
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+typedef struct BufferSyncPartition
+{
+ int prev_strategy_buf_id;
+ uint32 prev_strategy_passes;
+ int next_to_clean;
+ uint32 next_passes;
+} BufferSyncPartition;
+
+static BufferSyncPartition *saved_info = NULL;
+static bool saved_info_valid = false;
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition,
+ BufferSyncPartition *saved)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3880,25 +3881,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3910,17 +3902,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - saved->prev_strategy_passes;
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - saved->prev_strategy_buf_id;
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (saved->next_passes - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - saved->next_to_clean;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3928,11 +3920,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (saved->next_passes == strategy_passes &&
+ saved->next_to_clean >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (saved->next_to_clean - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3952,9 +3944,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3968,15 +3960,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ saved->prev_strategy_buf_id = strategy_buf_id;
+ saved->prev_strategy_passes = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3984,9 +3977,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3996,7 +3989,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -4004,10 +3997,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -4034,7 +4027,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -4059,20 +4052,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(saved->next_to_clean, true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++saved->next_to_clean >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ saved->next_to_clean = first_buffer;
+ saved->next_passes++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -4086,7 +4079,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -4116,8 +4109,83 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ /* allocate space for per-partition information between calls */
+ if (saved_info == NULL)
+ {
+ /*
+ * XXX Not great it's using malloc(), but how else to allocate a
+ * variable-length array?
+ */
+ saved_info = malloc(sizeof(BufferSyncPartition) * num_partitions);
+ }
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition,
+ &saved_info[partition]);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 53ef5239e8d..2d56579682e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -36,17 +36,28 @@
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock sweep partition */
+ int32 node;
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -57,11 +68,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+
+ /* cached info about freelist partitioning */
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -108,6 +140,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -119,6 +152,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -126,14 +160,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -159,19 +193,118 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /*
+ * Make sure we've calculated a buffer in the range of the partition. Buffer
+ * IDs are 1-based, we're calculating 0-based indexes.
+ */
+ Assert((victim >= 0) && (victim < sweep->numBuffers));
+ Assert(BufferIsValid(1 + sweep->firstBuffer + victim));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * ClockSweepPartitionIndex
+ * pick the clock-sweep partition to use based on PID and NUMA node
+ *
+ * With libnuma, use the NUMA node and PID to pick the partition. Otherwise
+ * use just PID (as if there's a single NUMA node).
+ *
+ * XXX This should also check if buffers are NUMA-partitioned, not just if
+ * compiled with libnuma.
+ */
+static int
+ClockSweepPartitionIndex(void)
+{
+ int node = 0,
+ index;
+ pid_t pid = MyProcPid;;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * If buffers are NUMA-partitioned, determine the partition using the NUMA
+ * node and PID. Without NUMA assume everything is a single NUMA node 0, and
+ * we pick the partition based on PID.
+ */
+#ifdef USE_LIBNUMA
+ if (shared_buffers_numa)
+ {
+ int cpu;
+
+ /* XXX do we need to check sched_getcpu is available, somehow? */
+ if ((cpu = sched_getcpu()) < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+ }
+#endif
+
+ /*
+ * We should't get unexpected NUMA nodes, not considered when setting up the
+ * buffer partitions. It could happen if the allowed NUMA nodes get adjusted
+ * at runtime, but at this point we just create partitions for all existing
+ * nodes. We could plan for allowed partitions, but then what if those get
+ * disabled, and the user allows some other partitions?
+ */
+ if ((node < 0) || (node > StrategyControl->num_nodes))
+ elog(ERROR, "node out of range: %d > %u", node, StrategyControl->num_nodes);
+
+ /*
+ * Calculate the partition index. Nodes have the same number of partitions,
+ * and we use the PID to pick one of those (for a given node). If there's
+ * only a single partition per node, we can ignore PID and use node directly.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = node;
+ }
+ else
+ {
+ /* use PID to pick one of node's partitions */
+ index = (node * StrategyControl->num_partitions_per_node)
+ + (pid % StrategyControl->num_partitions_per_node);
+ }
+
+ /* should have a valid partition index */
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and PID
+ *
+ * Pick a partition mapped to the NUMA node the backend is currently running
+ * on, and use PID if there are multiple partitions per node. Without NUMA
+ * supported/enabled, use just PID.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = ClockSweepPartitionIndex();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -242,10 +375,37 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* We count buffer allocation requests so that the bgwriter can estimate
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
+ *
+ * XXX It's not quite right we call ChooseClockSweep twice - now, and then
+ * a couple lines later (through ClockSweepTick). If the process moves
+ * between CPUs / NUMA nodes in between, these call may pick different
+ * partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * a process "sweeps" only a fraction of buffers, even if the other buffers
+ * are better candidates for eviction. Maybe there should be some logic to
+ * "steal" buffers from other partitions or other nodes?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Needs to scan partitions if needed, as a fallback.
+ *
+ * XXX Would that also mean we should have multiple bgwriters, one for each
+ * node, or would one bgwriter still handle all nodes?
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -325,6 +485,48 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* XXX Do we need the lock, if we're only accessing atomics? Surely not. */
+ /* XXX Are we ever calling this without num_buf_alloc? */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -332,37 +534,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -394,8 +603,14 @@ StrategyNotifyBgWriter(int bgwprocno)
static void
StrategyCtlShmemRequest(void *arg)
{
+ int num_partitions;
+
+ /* get the number of buffer partitions */
+ BufferPartitionsCalculate(NULL, &num_partitions, NULL);
+
ShmemRequestStruct(.name = "Buffer Strategy Status",
- .size = sizeof(BufferStrategyControl),
+ .size = offsetof(BufferStrategyControl, sweeps) +
+ mul_size(num_partitions, sizeof(ClockSweep)),
.ptr = (void **) &StrategyControl
);
}
@@ -408,12 +623,42 @@ StrategyCtlShmemInit(void *arg)
{
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Remember the number of partitions */
+ BufferPartitionsParams(&StrategyControl->num_nodes,
+ &StrategyControl->num_partitions,
+ &StrategyControl->num_partitions_per_node);
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].node = node;
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -777,3 +1022,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e944cee2e91..5ab0cee4281 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,7 +593,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
/* buf_table.c */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 1cf09e8fb7c..e0bb4cc1df1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -410,6 +410,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index ae977297849..f68e08e5697 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 45fc91fcb97..a4617481f7c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -451,6 +451,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.54.0
[text/x-patch] v20260624-0005-clock-sweep-balancing-of-allocations.patch (27.4K, ../../[email protected]/6-v20260624-0005-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From 4ed4cf8669a6f379ebbf39a2283704931d4b3279 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:28:10 +0200
Subject: [PATCH v20260624 5/7] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 428 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 471 insertions(+), 21 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 92176fed7f8..43d2e84f9d2 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -20,7 +20,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index b07fafda0d9..f26b2332c1d 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -31,7 +33,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -940,6 +942,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -971,6 +975,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -992,11 +1002,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -1005,8 +1021,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -1035,6 +1059,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7bedaf8987a..61be05a68d4 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4135,6 +4135,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 2d56579682e..a543fb12b21 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -35,6 +35,26 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX We need to make ClockSweep fixed-size, so that we can have an array
+ * in shared memory. The easiest way is to pick a sufficiently high value
+ * that no system will actually need. 32 seems high enough.
+ *
+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average, so that we don't flap too much. The higher the value, the
+ * more the old value affects the result.
+ *
+ * XXX Doesn't this obscure the interpretation of weights as probabilities to
+ * allocate from a given partition? Does it still sum to 100%? I don't think
+ * so, it's just a fraction of allocations to go from a given partition.
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -68,9 +88,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that partition. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * Backends use the budget from it's "home" partition, so that a busy
+ * partitions (with a lot of processes on that NUMA node etc.) spread
+ * the allocations evenly.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -140,7 +183,66 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets a
+ * budget for allocating buffers from other partitions. A process that
+ * "exhausts" a budget in it's home partition gets redirected to the other
+ * partitions, driven by the budgets.
+ *
+ * For example, a partition may have budget [25, 25, 25, 25], which means
+ * each of the 4 partitions should get 1/4 of allocations. Or the buget
+ * can be [50, 50, 0, 0], which means all allocations will go to the first
+ * two partitions (one of them being the "home" one);
+ *
+ * We could do that based on a random number generator, but for now we
+ * simply treat the values as a budget, i.e. a number of allocations to
+ * serve from other partitions, and move in round-robin way.
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balanced. We can make the budgets higher
+ * (say, to match the expected number of allocations, i.e. bout the average
+ * number of allocations from the past interval). Or maybe configurable.
+ *
+ * XXX We should always start allocating from the "home" partition, i.e.
+ * from from it, and only then redirect to other partitions.
+ *
+ * XXX It probably is not great all the processes from that "home"
+ * partition are coordinated, and move to between partitions at about the
+ * same time. Not sure what to do about this.
+ *
+ * XXX We should also prefer other partitions from the same NUMA node (if
+ * there are some). Probably by setting the budgets.
+ *
+ * FIXME Explain at which point are the budgets recalculated, by which
+ * process, and how that affects other processes allocating buffers.
+ */
+
+/*
+ * The "optimal" clock-sweep partition. After a backend gets moved to a
+ * different NUMA node, we restart the balancing so that it uses the
+ * correct "budget" from the new home partition.
+ */
+static int clocksweep_partition_home = -1;
+
+/*
+ * The partition the backend is currently allocating from (either the
+ * home one, or one of the redirected ones).
+ */
+static int clocksweep_partition_current = -1;
+
+/*
+ * The number of buffers to allocate from the current partition.
+ */
+static int clocksweep_partition_budget = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -152,7 +254,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -300,11 +402,68 @@ ClockSweepPartitionIndex(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
+ /* What's the "optimal" partition for this backend? */
int index = ClockSweepPartitionIndex();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Was the process migrated to a different NUMA node? If the home partition
+ * changed, we need to reset the budget and start over, so that we correctly
+ * prefer "nearby" partitions etc.
+ *
+ * XXX Could this be a problem when processes move all the time? I don't
+ * think so - if a process moves between many partitions, that alone will
+ * spread the allocations over partitions. Similarly, if there are many
+ * processes, that should make it even more even.
+ */
+ if (clocksweep_partition_home != index)
+ {
+ clocksweep_partition_home = index;
+ clocksweep_partition_current = index;
+ clocksweep_partition_budget = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_home != -1);
+ Assert(clocksweep_partition_current != -1);
+ Assert(clocksweep_partition_budget >= 0);
+
+ /*
+ * When balancing allocations, redirect the allocations to other partitions
+ * according to the budgets. We move through partitions in a round-robin way,
+ * after allocating the "budget" of allocations from the current one.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of budget from the current partition? Move to the next one
+ * with non-zero budget.
+ */
+ while (clocksweep_partition_budget == 0)
+ {
+ /* wrap around at the end */
+ clocksweep_partition_current++;
+ if (clocksweep_partition_current >= StrategyControl->num_partitions)
+ clocksweep_partition_current = 0;
+
+ clocksweep_partition_budget
+ = sweep->balance[clocksweep_partition_current];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the current allocation */
+ --clocksweep_partition_budget;
+
+ /*
+ * Account for the allocation in the "home" partition, so that the next
+ * round of rebalancing (recalculating the budgets) knows about the
+ * allocation traffic in various partitions.
+ */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition_current];
}
/*
@@ -381,7 +540,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* between CPUs / NUMA nodes in between, these call may pick different
* partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -485,6 +644,229 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition budgets, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * This means an allocation may be requested in partition A (i.e. the
+ * home partition of the process requesting it), but end up allocating
+ * the buffer in partition B. We have a counter for both - the number of
+ * allocations requested in a partition, and the number of allocations
+ * actually handled by that partition. The former is used for calculating
+ * weights, the latter is used only for monitoring.
+ *
+ * The balancing happens in intervals - it adjusts future allocations
+ * based on stats about recent allocations, namely:
+ *
+ * - numBufferAllocs - number of allocations served by a partition
+ *
+ * - numRequestedAllocs - number of allocatios requested in a partition
+ *
+ * We're trying to smooth numBufferAllocs in the next interval, based on
+ * numRequestedAllocs measured in the last interval.
+ *
+ * The balancing algorithm works like this:
+ *
+ * - the target (average number of allocations per partition) is calculated
+ * from total number of allocations requested in the last intervaal
+ *
+ * - partitions get divided into two groups - those with more allocation
+ * requests than the target, and those with fewer requests
+ *
+ * - we "distribute" the delta (which is the same between the groups)
+ * between the groups (one has more, the other fewer)
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX Currently this does not give preference to other partitions on the
+ * same NUMA node (redirect to it first), but it could.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocation requests for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * XXX We lock the partitions one by one, so this is not a perfectly
+ * consistent snapshot of the counts, and the resets happen before we
+ * update the weights too. But we're only looking for heuristics, so
+ * this should be good enough.
+ *
+ * XXX A similar issue applies to the counter reset later - we haven't
+ * updated the weights yet, so some of the requests counted for the next
+ * interval will be redirected per current weights. Should be fine, it's
+ * just an approximate heuristics, and there should be very few requests in
+ * between. Alternatively, we could reset the request counters when setting
+ * the new weights, and just ignore the couple requests in between.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /* Calculate the "fair share" of allocations per partition. */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state for each partition, i.e. how
+ * many more/fewer allocations it handled relative to the average.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip rebalancing when there's not enough activity, and just keep the
+ * current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary. And maybe we should
+ * do the rabalancing in this case anyway, it's likely cheap and on a big
+ * system 10% can be quite a lot.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * The actual rebalancing
+ *
+ * Partition with fewer than average allocations, should not redirect any
+ * allocations to other partitions. So just use weights with a single
+ * non-zero weight for the partition itself.
+ *
+ * Partition with more than average allocations, should not receive any
+ * redirected allocations, and instead it should redirect excess allocations
+ * to other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of a
+ * partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from it.
+ *
+ * XXX We should add hysteresis, so that it does not oscillate or something
+ * like that. Maybe CLOCKSWEEP_HISTORY_COEFF already does that?
+ *
+ * XXX Ideally, the alternative partitions to use first would be the other
+ * partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -657,7 +1039,21 @@ StrategyCtlShmemInit(void *arg)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1025,8 +1421,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1034,11 +1432,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5ab0cee4281..887314d43f4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,6 +593,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e0bb4cc1df1..02833b19b0c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -411,11 +411,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.54.0
[text/x-patch] v20260624-0006-clock-sweep-scan-all-partitions.patch (6.2K, ../../[email protected]/7-v20260624-0006-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From fafa9ba54c0201cec1084f558cdad285b5fc903b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:39:38 +0200
Subject: [PATCH v20260624 6/7] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 83 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index a543fb12b21..1ac1e3e3490 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -184,6 +184,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint64 *buf_state);
/*
* clocksweep allocation balancing
@@ -251,10 +254,9 @@ static int clocksweep_partition_budget = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -486,7 +488,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -545,33 +548,61 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * a process "sweeps" only a fraction of buffers, even if the other buffers
- * are better candidates for eviction. Maybe there should be some logic to
- * "steal" buffers from other partitions or other nodes?
- *
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Needs to scan partitions if needed, as a fallback.
- *
- * XXX Would that also mean we should have multiple bgwriters, one for each
- * node, or would one bgwriter still handle all nodes?
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint64 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint64 old_buf_state;
uint64 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -599,7 +630,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -618,7 +649,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index f68e08e5697..ae977297849 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.54.0
[text/x-patch] v20260624-0007-Add-parttioned-clocksweep-and-NUMA-goodies.patch (14.6K, ../../[email protected]/8-v20260624-0007-Add-parttioned-clocksweep-and-NUMA-goodies.patch)
download | inline diff:
From f5b7b6733419186e56b06ed1d8b6da7e34984c2f Mon Sep 17 00:00:00 2001
From: Jakub Wartak <[email protected]>
Date: Thu, 11 Jun 2026 12:38:43 +0200
Subject: [PATCH v20260624 7/7] Add parttioned clocksweep and NUMA goodies.
1. Add three clocksweep GUCs to allow manipulation of partitoned clocksweep
in runtime.
2. Add pg_buffercache_set_weights(int, int[]) to alter partition allocs (with
clocksweep_balance_recalc=off).
3. Add debug_numa_node GUC to pin to NUMA node.
---
.../pg_buffercache--1.7--1.8.sql | 7 ++
contrib/pg_buffercache/pg_buffercache_pages.c | 37 ++++++++
src/backend/storage/buffer/freelist.c | 86 ++++++++++++++++++-
src/backend/tcop/postgres.c | 57 ++++++++++++
src/backend/utils/misc/guc_parameters.dat | 31 +++++++
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 6 ++
7 files changed, 224 insertions(+), 1 deletion(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 43d2e84f9d2..9d8f4969555 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -25,9 +25,16 @@ CREATE VIEW pg_buffercache_partitions AS
num_req_allocs bigint, -- handled allocs (current cycle)
weights int[]); -- balancing weights
+-- Register the function to set clock-sweep balance weights.
+CREATE FUNCTION pg_buffercache_set_partition(IN partition int, IN weights int[])
+RETURNS void
+AS 'MODULE_PATHNAME', 'pg_buffercache_set_partition'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+REVOKE ALL ON FUNCTION pg_buffercache_set_partition(int, int[]) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index f26b2332c1d..2a1b7cb7549 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -81,6 +81,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_set_partition);
/* Only need to touch memory once per backend process lifetime */
@@ -1077,3 +1078,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Set the clock-sweep balance weights for a single partition.
+ */
+Datum
+pg_buffercache_set_partition(PG_FUNCTION_ARGS)
+{
+ int partition = PG_GETARG_INT32(0);
+ ArrayType *array = PG_GETARG_ARRAYTYPE_P(1);
+ Datum *elems;
+ bool *nulls;
+ int nelems;
+ int *weights;
+
+ if (ARR_NDIM(array) > 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("weights must be a one-dimensional array")));
+
+ deconstruct_array_builtin(array, INT4OID, &elems, &nulls, &nelems);
+
+ weights = palloc_array(int, nelems);
+ for (int i = 0; i < nelems; i++)
+ {
+ if (nulls[i])
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("weights must not contain NULL values")));
+
+ weights[i] = DatumGetInt32(elems[i]);
+ }
+
+ ClockSweepSetWeights(partition, weights, nelems);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 1ac1e3e3490..e677c71e0b3 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -55,6 +55,26 @@
*/
#define CLOCKSWEEP_HISTORY_COEFF 0.5
+/*
+ * GUCs controlling the NUMA-aware clock-sweep behavior.
+ *
+ * clocksweep_balance - when enabled, allocations may get redirected between
+ * clock-sweep partitions to keep them balanced (see StrategySyncBalance).
+ *
+ * clocksweep_balance_recalc - when enabled, the balance weights are
+ * periodically recalculated (see StrategySyncBalance). Disabling this keeps
+ * the current weights, e.g. ones configured manually using
+ * pg_buffercache_set_partition(), while still using them to balance
+ * allocations (if clocksweep_balance is enabled).
+ *
+ * clocksweep_scan_all_partitions - when enabled, looking for a free buffer
+ * scans all clock-sweep partitions (in a round-robin way), not just the
+ * backend's "home" partition.
+ */
+bool clocksweep_balance = true;
+bool clocksweep_balance_recalc = true;
+bool clocksweep_scan_all_partitions = true;
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -436,8 +456,11 @@ ChooseClockSweep(bool balance)
* When balancing allocations, redirect the allocations to other partitions
* according to the budgets. We move through partitions in a round-robin way,
* after allocating the "budget" of allocations from the current one.
+ *
+ * Balancing can be disabled at runtime using the clocksweep_balance GUC, in
+ * which case allocations always stay in the backend's "home" partition.
*/
- if (balance)
+ if (balance && clocksweep_balance)
{
/*
* Ran out of budget from the current partition? Move to the next one
@@ -551,6 +574,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* Start with the "preferred" partition, and then proceed in a round-robin
* manner. If we cycle back to the starting partition, it means none of the
* partitions has unpinned buffers.
+ *
+ * Scanning of the other partitions can be disabled at runtime using the
+ * clocksweep_scan_all_partitions GUC. In that case we only scan the
+ * backend's "home" partition, and fail if it has no unpinned buffers.
*/
sweep = ChooseClockSweep(true);
sweep_start = sweep;
@@ -562,6 +589,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
if (buf != NULL)
return buf;
+ /* don't look at other partitions unless allowed to */
+ if (!clocksweep_scan_all_partitions)
+ break;
+
/*
* Try advancing to the next partition, round-robin (if last partition,
* wrap around to the beginning).
@@ -739,6 +770,9 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ if (!clocksweep_balance || !clocksweep_balance_recalc)
+ return;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -1481,3 +1515,53 @@ ClockSweepPartitionGetInfo(int idx,
(*weights)[i] = (int) sweep->balance[i];
}
}
+
+/*
+ * ClockSweepSetWeights override the clock-sweep balance weights of a single
+ * partition.
+ */
+void
+ClockSweepSetWeights(int partition, int *weights, int nweights)
+{
+ ClockSweep *sweep;
+
+ /*
+ * Disallow manual weights while the automatic recalculation is enabled, as
+ * StrategySyncBalance would just recompute (and overwrite) them.
+ */
+ if (clocksweep_balance_recalc)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set clock-sweep weights while debug_clocksweep_balance_recalc is enabled"),
+ errhint("Set debug_clocksweep_balance_recalc to off before setting the weights manually.")));
+
+ if ((partition < 0) || (partition >= StrategyControl->num_partitions))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("clock-sweep partition %d out of range", partition),
+ errhint("There are %d clock-sweep partitions, numbered from 0.",
+ StrategyControl->num_partitions)));
+
+ if (nweights != StrategyControl->num_partitions)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("number of weights (%d) does not match number of clock-sweep partitions (%d)",
+ nweights, StrategyControl->num_partitions)));
+
+ for (int i = 0; i < nweights; i++)
+ {
+ if ((weights[i] < 0) || (weights[i] > 100))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("clock-sweep weight %d out of range",
+ weights[i]),
+ errhint("Each weight must be between 0 and 100.")));
+ }
+
+ sweep = &StrategyControl->sweeps[partition];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ for (int j = 0; j < nweights; j++)
+ sweep->balance[j] = (uint8) weights[j];
+ SpinLockRelease(&sweep->clock_sweep_lock);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index dbef734a93f..02787062c83 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,7 @@
#include "utils/timeout.h"
#include "utils/timestamp.h"
#include "utils/varlena.h"
+#include <numa.h>
/* ----------------
* global variables
@@ -110,6 +111,9 @@ int client_connection_check_interval = 0;
/* flags for non-system relation kinds to restrict use */
int restrict_nonsystem_relation_kind;
+/* NUMA node to pin the backend to at query start; -1 disables pinning */
+int debug_numa_node = -1;
+
/*
* Include signal sender PID/UID in the server log when available
* (SA_SIGINFO). The caller must supply the already-captured pid and uid
@@ -1020,6 +1024,53 @@ pg_plan_queries(List *querytrees, const char *query_string, int cursorOptions,
}
+/*
+ * process_debug_numa_node
+ *
+ * If the debug_numa_node GUC is set (>= 0), pin this backend to run on the
+ * CPUs of the requested NUMA node. -1 disables it (default)
+ */
+static void
+process_debug_numa_node(void)
+{
+#ifdef USE_LIBNUMA
+ static int applied_numa_node = -1;
+
+ if (debug_numa_node == applied_numa_node)
+ return;
+
+ /* Nothing we can do if the kernel/library has no NUMA support. */
+ if (numa_available() < 0)
+ {
+ applied_numa_node = debug_numa_node;
+ return;
+ }
+
+ if (debug_numa_node < 0)
+ {
+ /* Pinning disabled: allow running on all nodes again. */
+ numa_run_on_node_mask(numa_all_nodes_ptr);
+ }
+ else if (debug_numa_node > numa_max_node())
+ {
+ ereport(WARNING,
+ (errmsg("debug_numa_node %d exceeds the highest available NUMA node %d, ignoring",
+ debug_numa_node, numa_max_node())));
+ }
+ else if (numa_run_on_node(debug_numa_node) != 0)
+ {
+ ereport(WARNING,
+ (errmsg("could not pin backend to NUMA node %d: %m",
+ debug_numa_node)));
+ }
+ else
+ elog(DEBUG1, "pinned backend to NUMA node %d", debug_numa_node);
+
+ applied_numa_node = debug_numa_node;
+#endif
+}
+
+
/*
* exec_simple_query
*
@@ -1044,6 +1095,9 @@ exec_simple_query(const char *query_string)
pgstat_report_activity(STATE_RUNNING, query_string);
+ /* Pin the backend to the configured NUMA node, if requested. */
+ process_debug_numa_node();
+
TRACE_POSTGRESQL_QUERY_START(query_string);
/*
@@ -2185,6 +2239,9 @@ exec_execute_message(const char *portal_name, long max_rows)
pgstat_report_activity(STATE_RUNNING, sourceText);
+ /* Pin the backend to the configured NUMA node, if requested. */
+ process_debug_numa_node();
+
foreach(lc, portal->stmts)
{
PlannedStmt *stmt = lfirst_node(PlannedStmt, lc);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 2e71c04282c..b72fce4b92b 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -641,6 +641,27 @@
boot_val => 'DEFAULT_ASSERT_ENABLED',
},
+{ name => 'debug_clocksweep_balance', type => 'bool', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Enables balancing of buffer allocations between clock-sweep partitions.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'clocksweep_balance',
+ boot_val => 'true'
+},
+
+{ name => 'debug_clocksweep_balance_recalc', type => 'bool', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Enables periodic recalculation of clock-sweep partition balance weights.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'clocksweep_balance_recalc',
+ boot_val => 'true'
+},
+
+{ name => 'debug_clocksweep_scan_all_partitions', type => 'bool', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Enables scanning all clock-sweep partitions when looking for a free buffer.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'clocksweep_scan_all_partitions',
+ boot_val => 'true'
+},
+
{ name => 'debug_copy_parse_plan_trees', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Set this to force all parse and plan trees to be passed through copyObject(), to facilitate catching errors and omissions in copyObject().',
flags => 'GUC_NOT_IN_SAMPLE',
@@ -693,6 +714,16 @@
options => 'debug_logical_replication_streaming_options',
},
+{ name => 'debug_numa_node', type => 'int', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Pins the backend to the given NUMA node at query start.',
+ long_desc => '-1 (the default) disables pinning.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_node',
+ boot_val => '-1',
+ min => '-1',
+ max => 'INT_MAX',
+},
+
{ name => 'debug_parallel_query', type => 'enum', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Forces the planner\'s use parallel query nodes.',
long_desc => 'This can be useful for testing the parallel query infrastructure by forcing the planner to generate plans that contain nodes that perform tuple communication between workers and the main process.',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index bf38aa6baa2..9590f7ac467 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -215,6 +215,7 @@ extern PGDLLIMPORT bool MyDatabaseHasLoginEventTriggers;
extern PGDLLIMPORT bool shmem_populate;
extern PGDLLIMPORT bool shmem_interleave;
+extern PGDLLIMPORT int debug_numa_node;
/*
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 02833b19b0c..b5c3873acae 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -190,6 +190,11 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
+/* in freelist.c */
+extern PGDLLIMPORT bool clocksweep_balance;
+extern PGDLLIMPORT bool clocksweep_balance_recalc;
+extern PGDLLIMPORT bool clocksweep_scan_all_partitions;
+
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
extern PGDLLIMPORT int effective_io_concurrency;
@@ -418,6 +423,7 @@ extern void ClockSweepPartitionGetInfo(int idx,
uint64 *buffer_total_req_allocs,
uint32 *buffer_req_allocs,
int **weights);
+extern void ClockSweepSetWeights(int partition, int *weights, int nweights);
/* inline functions */
--
2.54.0
[application/x-compressed-tar] numa-scripts.tgz (1.2K, ../../[email protected]/9-numa-scripts.tgz)
download
[application/pdf] numa- median latency.pdf (37.5K, ../../[email protected]/10-numa-%20median%20latency.pdf)
download
[application/pdf] numa - tps.pdf (38.6K, ../../[email protected]/11-numa%20-%20tps.pdf)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-25 12:19 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2026-06-25 12:19 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 24, 2026 at 10:29 PM Tomas Vondra <[email protected]> wrote:
Hi,
> Here's an updated patch series, with only minor changes to fix the mbind
> issues:
[..]
> I've also included Jakub's "goodies" patch with the additional GUCs.
> Those seem potentially useful to development.
Cool!
> I have some results from a new round of benchmarks, and it's a bit
> disappointing. Or rather, there seem to be some issues that I can't
> figure out, causing regressions.
[..]
> This chart is for median latency (in milliseconds):
>
> clients master 0003 0004 0003/on 0004/on
> -------------------------------------------------------------
> 1 12767 12582 14509 12807 15307
> 8 14383 14355 14149 14069 16165
> 32 14756 15198 14836 14984 17128
> --------------------------------------------------------
> 1 103% 114% 100% 120%
> 8 101% 98% 98% 112%
> 32 102% 101% 102% 116%
>
I haven't tried it yet, however I can spot some things:
No crystal clear idea why, but in the script I can see that you have
io_method = io_uring and are not dropping_caches, so IMHO it is too complex
interaction at this stage.
One hint: such setup is going to be problematic for proving numbers.
On the meeting I've tried to describe that I've been using io_method = sync
instead of 'worker' to get more predicitable results (together with echo 3
> drop_caches), because then it is that backend's CPU/$NODE doing that
pread()/pwrite() -- or any other operating performing the load --
it is going to put that file onto that_specific_$NODE --
so even if you have sequence like:
pgbench -i
pg_ctl restart
pgbench -c XX
then pgbench -i even with shared_buffers_numa=on will spread into many
nodes the Buffers, yet after the restart the VFS cache portion of the data
will still reside on single specific $NODE that wrote it to the filesystem
(due to local-first-tocuh-affinity even for VFS cache), so any further reading:
VFS cache --pread()--> s_b will take the hit of remote interconnect with
some probablity depending on where the new backends are running. Also
with worker it is even worse as we have those memory queue in between. I
think we even can have this:
file in VFS cache @ node0 --because of first touch policy (pgbench -i/prewarm)
io worker @ node1 --hits latency from node0 and node2
shm io worker queue @ node2 --well
client backend @ node 3 --puts into shm io worker on node2
Therefore I'm sticking to 'sync' to ease the pain... but with uring, I suspect
the situation is kind of similiar as we call io_uring_submit(), and we
may endup using io-wq kernel threads, and we have those submission/receive
(memory) queues that are located somewhere (that is on some node) too.
I think, we simply lack affinity for IO/NUMA for all io modes except sync, but
it's too early I suspect and way outside of scope for this $thread. I've
started thinking about it just last week, so... (but hopefully I'll be able
to ship helper fscachenuma.c to show layout of file across VFS caches on nodes
next week I hope)
Maybe some other suggestions:
Q1) Maybe some crosschecks first?
# balance should be equal between nodes even for baseline
# linux kernel has tendency to fit shm into one if it fits
find /sys/devices/system/node*/ -name 'free_hugepages' -exec
grep -H . {} \;
# check N0 and N1 even for default policy, might also reveal imbalance
# lots of RAM and too big huge_pages allows fitting whole shm
into just N0
# see point 4 from [1]
grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps
# then during pgbench -c run maybe those:
mpstat -N ALL 1
perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
--per-socket -I 1000 # or -M
memory_bandwidth_read,memory_bandwidth_write
(it might reveal that problem I've described above about io_method:
even with pgbench -c 1 you might be reading from all sockets/wrong sockets
instead of the correct one with affinity)
I like to pin CPUs to just one node for pgbench -c
<NUMBER_OF_CPUS/NUMBER_OF_NODES>
[to saturate one node only] and start server also with CPU pining
[or use this debug_numa_node to force] to that one node and cross-check
what's being read (using perf) and usually I have to disarm clock balancing
and override weights using pg_buffercache_set_partition() to also force
weight to stay local only - only then I'm able to outrun master. That's
how this idea was born that if we are only working on node $N with
some relations
then let's use only node $N's Buffers. But I have 90us:~280us
local vs remote
latency, so it's probably way easier for me to see results even without
disabling CPU-idle-states/turboboost/etc.
Q2) Dunno, but 0007 is not changing anything in runtime and you get huge
discrepeancy results when going 0006 -> 0007 for clients=1 (see
128% -> 112%).
Literally, as the same code but different rebuild (ELF image)
would be having
vastly different layout enough to cause perf issues?
Hopefully next week I'll try to repro those numbers to see if I can
help more.
-J.
[1] - https://www.postgresql.org/message-id/CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay%2BmOEKs9rzCkegUA%40mail.g...
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-25 13:49 Tomas Vondra <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Tomas Vondra @ 2026-06-25 13:49 UTC (permalink / raw)
To: Jakub Wartak <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On 6/25/26 14:19, Jakub Wartak wrote:
> On Wed, Jun 24, 2026 at 10:29 PM Tomas Vondra <[email protected]> wrote:
>
> Hi,
>
>> Here's an updated patch series, with only minor changes to fix the mbind
>> issues:
> [..]
>> I've also included Jakub's "goodies" patch with the additional GUCs.
>> Those seem potentially useful to development.
>
> Cool!
>
>> I have some results from a new round of benchmarks, and it's a bit
>> disappointing. Or rather, there seem to be some issues that I can't
>> figure out, causing regressions.
> [..]
>> This chart is for median latency (in milliseconds):
>>
>> clients master 0003 0004 0003/on 0004/on
>> -------------------------------------------------------------
>> 1 12767 12582 14509 12807 15307
>> 8 14383 14355 14149 14069 16165
>> 32 14756 15198 14836 14984 17128
>> --------------------------------------------------------
>> 1 103% 114% 100% 120%
>> 8 101% 98% 98% 112%
>> 32 102% 101% 102% 116%
>>
>
> I haven't tried it yet, however I can spot some things:
>
> No crystal clear idea why, but in the script I can see that you have
> io_method = io_uring and are not dropping_caches, so IMHO it is too complex
> interaction at this stage.
>
By caches I assume you mean page cache? The test is meant so simulate a
cached system, copying data between shared buffers and page cache. My
expectation is that once we start hitting I/O, it'll completely hide
most differences due to NUMA.
> One hint: such setup is going to be problematic for proving numbers.
> On the meeting I've tried to describe that I've been using io_method = sync
> instead of 'worker' to get more predicitable results (together with echo 3
>> drop_caches), because then it is that backend's CPU/$NODE doing that
> pread()/pwrite() -- or any other operating performing the load --
> it is going to put that file onto that_specific_$NODE --
> so even if you have sequence like:
> pgbench -i
> pg_ctl restart
> pgbench -c XX
>
Hmm, I missed that point during the meeting. I wonder if "worker" is
interacting with the NUMA somehow (I mean, does it load it into the
right node?). But I'm using io_uring, and it's not clear to me why sync
would be better for benchmarking?
Ultimately, we need to make sure it works well with io_uring anyway,
right? Even if "sync" happens to be better for benchmarking (or even for
NUMA stuff), we have to make it work with worker/io_uring. Because
that's what practical systems use.
> then pgbench -i even with shared_buffers_numa=on will spread into many
> nodes the Buffers, yet after the restart the VFS cache portion of the data
> will still reside on single specific $NODE that wrote it to the filesystem
> (due to local-first-tocuh-affinity even for VFS cache), so any further reading:
> VFS cache --pread()--> s_b will take the hit of remote interconnect with
> some probablity depending on where the new backends are running. Also
> with worker it is even worse as we have those memory queue in between. I
> think we even can have this:
>
> file in VFS cache @ node0 --because of first touch policy (pgbench -i/prewarm)
> io worker @ node1 --hits latency from node0 and node2
> shm io worker queue @ node2 --well
> client backend @ node 3 --puts into shm io worker on node2
>
> Therefore I'm sticking to 'sync' to ease the pain... but with uring, I suspect
> the situation is kind of similiar as we call io_uring_submit(), and we
> may endup using io-wq kernel threads, and we have those submission/receive
> (memory) queues that are located somewhere (that is on some node) too.
>
> I think, we simply lack affinity for IO/NUMA for all io modes except sync, but
> it's too early I suspect and way outside of scope for this $thread. I've
> started thinking about it just last week, so... (but hopefully I'll be able
> to ship helper fscachenuma.c to show layout of file across VFS caches on nodes
> next week I hope)
>
Ah, you're suggesting the page cache stuff will be placed on a single
NUMA node? That may be true, it's a good point. And maybe it could skew
the results in a bad way. Still, that would be the case even without the
NUMA partitioning, no?
> Maybe some other suggestions:
>
> Q1) Maybe some crosschecks first?
> # balance should be equal between nodes even for baseline
> # linux kernel has tendency to fit shm into one if it fits
> find /sys/devices/system/node*/ -name 'free_hugepages' -exec
> grep -H . {} \;
>
> # check N0 and N1 even for default policy, might also reveal imbalance
> # lots of RAM and too big huge_pages allows fitting whole shm
> into just N0
> # see point 4 from [1]
> grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps
>
> # then during pgbench -c run maybe those:
> mpstat -N ALL 1
> perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
> --per-socket -I 1000 # or -M
> memory_bandwidth_read,memory_bandwidth_write
>
> (it might reveal that problem I've described above about io_method:
> even with pgbench -c 1 you might be reading from all sockets/wrong sockets
> instead of the correct one with affinity)
>
I'll try, but if you could try running some experiments on your own,
that might be helpful.
> I like to pin CPUs to just one node for pgbench -c
> <NUMBER_OF_CPUS/NUMBER_OF_NODES>
> [to saturate one node only] and start server also with CPU pining
> [or use this debug_numa_node to force] to that one node and cross-check
> what's being read (using perf) and usually I have to disarm clock balancing
> and override weights using pg_buffercache_set_partition() to also force
> weight to stay local only - only then I'm able to outrun master. That's
> how this idea was born that if we are only working on node $N with
> some relations
> then let's use only node $N's Buffers. But I have 90us:~280us
> local vs remote
> latency, so it's probably way easier for me to see results even without
> disabling CPU-idle-states/turboboost/etc.
>
> Q2) Dunno, but 0007 is not changing anything in runtime and you get huge
> discrepeancy results when going 0006 -> 0007 for clients=1 (see
> 128% -> 112%).
> Literally, as the same code but different rebuild (ELF image)
> would be having
> vastly different layout enough to cause perf issues?
>
> Hopefully next week I'll try to repro those numbers to see if I can
> help more.
>
Thank you! That'd be great.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-29 07:42 Jakub Wartak <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2026-06-29 07:42 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <[email protected]> wrote:
>
> >> I have some results from a new round of benchmarks, and it's a bit
> >> disappointing. Or rather, there seem to be some issues that I can't
> >> figure out, causing regressions.
> > [..]
> >> This chart is for median latency (in milliseconds):
> >>
> >> clients master 0003 0004 0003/on 0004/on
> >> -------------------------------------------------------------
> >> 1 12767 12582 14509 12807 15307
> >> 8 14383 14355 14149 14069 16165
> >> 32 14756 15198 14836 14984 17128
> >> --------------------------------------------------------
> >> 1 103% 114% 100% 120%
> >> 8 101% 98% 98% 112%
> >> 32 102% 101% 102% 116%
> >>
> >
> > I haven't tried it yet, however I can spot some things:
> >
> > No crystal clear idea why, but in the script I can see that you have
> > io_method = io_uring and are not dropping_caches, so IMHO it is too complex
> > interaction at this stage.
> >
>
> By caches I assume you mean page cache? The test is meant so simulate a
> cached system, copying data between shared buffers and page cache. My
> expectation is that once we start hitting I/O, it'll completely hide
> most differences due to NUMA.
No, it wont completley hide it, those differences at least here still matter
(AFAIR right now like +/- 10% here)
> > One hint: such setup is going to be problematic for proving numbers.
> > On the meeting I've tried to describe that I've been using io_method = sync
> > instead of 'worker' to get more predicitable results (together with echo 3
> >> drop_caches), because then it is that backend's CPU/$NODE doing that
> > pread()/pwrite() -- or any other operating performing the load --
> > it is going to put that file onto that_specific_$NODE --
> > so even if you have sequence like:
> > pgbench -i
> > pg_ctl restart
> > pgbench -c XX
> >
>
> Hmm, I missed that point during the meeting. I wonder if "worker" is
> interacting with the NUMA somehow (I mean, does it load it into the
> right node?). But I'm using io_uring, and it's not clear to me why sync
> would be better for benchmarking?
>
> Ultimately, we need to make sure it works well with io_uring anyway,
> right? Even if "sync" happens to be better for benchmarking (or even for
> NUMA stuff), we have to make it work with worker/io_uring. Because
> that's what practical systems use.
Yes, we need to make work with more advanced, but I don't think we are there
yet (we'll need some more patches in orde rto demonstrate it reliably).
> > then pgbench -i even with shared_buffers_numa=on will spread into many
> > nodes the Buffers, yet after the restart the VFS cache portion of the data
> > will still reside on single specific $NODE that wrote it to the filesystem
> > (due to local-first-tocuh-affinity even for VFS cache),
> > [.. blabla , use io_method=sync ]
> >
>
> Ah, you're suggesting the page cache stuff will be placed on a single
> NUMA node? That may be true, it's a good point. And maybe it could skew
> the results in a bad way.
I've just published [0], see for yourself:
This happens especiall after pgbench -i, so:
pgbench -i # pagecache placement on one NUMA node
pg_ctl restart
pgbench -c XX
is day and night different than let's say:
pgbench -i
echo 3 > drop_caches
pg_ctl restart
pgbench -c XX # pagecache placement happens by many backends
# potentially many NUMA nodes
> Still, that would be the case even without the NUMA partitioning, no?
Right, in my experience we should not benchmark against master started
with the default pg_ctl (that's is without numactl --interleave=all) because
it is confusing to reason about it due how the s_b could laid out without
that interleaving. I mean later we can switch to that default, but IMHO not
yet.
> > Maybe some other suggestions:
> >
> > Q1) Maybe some crosschecks first?
> > # balance should be equal between nodes even for baseline
> > # linux kernel has tendency to fit shm into one if it fits
> > find /sys/devices/system/node*/ -name 'free_hugepages' -exec
> > grep -H . {} \;
> >
> > # check N0 and N1 even for default policy, might also reveal imbalance
> > # lots of RAM and too big huge_pages allows fitting whole shm
> > into just N0
> > # see point 4 from [1]
> > grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps
> >
> > # then during pgbench -c run maybe those:
> > mpstat -N ALL 1
> > perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
> > --per-socket -I 1000 # or -M
> > memory_bandwidth_read,memory_bandwidth_write
> >
> > (it might reveal that problem I've described above about io_method:
> > even with pgbench -c 1 you might be reading from all sockets/wrong sockets
> > instead of the correct one with affinity)
> >
>
> I'll try, but if you could try running some experiments on your own,
> that might be helpful.
[..]
> > Hopefully next week I'll try to repro those numbers to see if I can
> > help more.
> >
>
> Thank you! That'd be great.
Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
that fscachenuma proggie to aid us in troubleshooting.
-J.
[0] - https://github.com/jakubwartakEDB/fscachenuma
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-06-30 12:51 Jakub Wartak <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 1 reply; 89+ messages in thread
From: Jakub Wartak @ 2026-06-30 12:51 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 29, 2026 at 9:42 AM Jakub Wartak
<[email protected]> wrote:
>
> On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <[email protected]> wrote:
> >
> > >> I have some results from a new round of benchmarks, and it's a bit
> > >> disappointing. Or rather, there seem to be some issues that I can't
> > >> figure out, causing regressions.
> > > [..]
> > >> This chart is for median latency (in milliseconds):
> > >>
> > >> clients master 0003 0004 0003/on 0004/on
> > >> -------------------------------------------------------------
> > >> 1 12767 12582 14509 12807 15307
> > >> 8 14383 14355 14149 14069 16165
> > >> 32 14756 15198 14836 14984 17128
> > >> --------------------------------------------------------
> > >> 1 103% 114% 100% 120%
> > >> 8 101% 98% 98% 112%
> > >> 32 102% 101% 102% 116%
> > >>
[..lots of variables..]
> > I'll try, but if you could try running some experiments on your own,
> > that might be helpful.
> [..]
> > > Hopefully next week I'll try to repro those numbers to see if I can
> > > help more.
> > >
> >
> > Thank you! That'd be great.
>
> Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
> that fscachenuma proggie to aid us in troubleshooting.
>
> -J.
>
> [0] - https://github.com/jakubwartakEDB/fscachenuma
Hi Tomas,
OK, so I've run couple of tests and modified run.sh and also tried to fix
some inefficiencies spotted while testing this. Note the attached
performance matrix is in TPS (so more is better). Raw results/CSV and
scripts are attached too.
* run2 = 2 workloads, partitioned pgbench_accounts
* run3 = just pgbenchS w/o partitioning + warmup
* run4 = semi-like pgbenchS w/o partitioning but 100k rows + warmup
One important modification in those run shell scripts is that they
clean page-cache (drop_cached) as mentioned earlier to avoid false results
where everything would on node#N after pgbench -i ran. Probably I did
not get any regressions you've got, because of this. Or better diff -u
run*.sh scripts.
The "inst-optimized" is just the same patchset (so "inst-patchset") + crude
attempt in 0008 to make further smooth out things and avoid regressions while
I've been working on this. 0008 does couple of things:
a. implements CPU/node caching instead quering it every single buffer. Even
if on x86_64 that is optimized by vdso/kernel to avoid the real syscall,
the semi-syscall tax seems to be visible when fetching lots of buffers.
128 is arbitrary and still kind of low (128*8kB=1MB, and we are doing
hundreths of MB/s; while rescheduling happened only every couple of
seconds).
b1. minimize the attempt to use other partittions till some threshold (
and then it relies on the scan-all-partitions)
b2. avoids selecting idle partitions (defined as avg_allocs/2) - if there
are low allocations there it is debatable if cache utilization is better
or sticking to lower latency is better (e.g. in some workloads buffer
reuse is close to 0, so lower latency is clearly better)
Results are attached, some observations:
0.There were vast differences in how pg_ctl is started (interleaved or not),
so I've decided in the end to show relative to both situations.
1.In run2/seqconcurrscans I've saturated my interconnect and that's why
it's giving 129-155% there. I don't have access physiscal hw, but I suspect
that modern 2socket EPYC5 has like ~614GB/s per socket RAM bandwidth,
but the max oneway bandwith of the interconnect is around ~220GB/s (
no way to provie it), so *IF* with hundreths of cores we would be able
fetch at this rate we could saturte modern hardware too that way (and
we birefly touched related topic: batched executor, accelerating it
so fast those effects could be more easily achieveable)
2.run3 has no partiitioning because according to perf and my eyes, it
spent time not on the buffers itself (thus it was way heavier on CPU
[partitioning] than on memory...), so that's how run3 was born without
partitions :D
3.The warmup is critical for run3/pgbenchS, as I've noticed that depending
on ${luck} if you start the "master" (baseline w/o interleaving) and pgbench
it right away everything might land on node0 (s_b, pagecache), so "master"
was basically cheating in benchmarks vs especially Your's patchset where
it was spreading way too soon. Having drop_caches, additional warump and
only then proper pgbench kind of reduces that luck-factor. In general I
think all runs with c=1 seem to have kind of low singal-to-noise ratio. I
was thinking about pinning to always stick to the same NUMA node from start
to win against master just for this c=1 scenarios, but "meh".
3b. in short for pgbench -S we can gain like 2-5%
4.run4 was made just to prove that workload fetching more buffers, than
the standard pgbench -S (1 row?), seems to be the key to prove
optimizations in 0008 (other than showing good benefits for seqconcurrscans
of course). So run4 just shows benefit compared to 0001-0007 alone.
Stil on the table:
1. maybe even better balancing is possible (?), but this one is seems enough?
I'm out of other ideas, well other than the
"shared-relation-use-by-foreign-node" idea described much earlier (but
I won't be able to pull that off), so I'm not entering this rabbit hole
any deeper.
2. Digging into io_method=worker optimizations (answering question: are they
necessary?) Maybe I'll throw in run5 quite soon, this is going to be
crucial to answer.
3. Potentially mentioned earlier BAS strategies (forcing just use of local
partitions for known-to-be-only-local-users: CTAS/VACCUM/etc), but I'm
afarid that's not for me as I would certainly break/violate some
invisible to me boundary.
Maybe You could run those run*.sh with master vs inst-patchset/optimized?
(I'm not sure, maybe there's even different factor at play too...)
-J.
Attachments:
[text/html] performance_report_run2.html (15.1K, ../../CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com/2-performance_report_run2.html)
download | inline:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Performance Evaluation Matrix</title>
<style>
body { font-family: 'Segoe UI', Arial, sans-serif; margin: 40px; color: #333; background-color: #fafafa; }
h2 { color: #2c3e50; border-bottom: 2px solid #bdc3c7; padding-bottom: 10px; margin-top: 0; }
.table-container { background: #fff; padding: 25px; border-radius: 6px; box-shadow: 0 2px 4px rgba(0,0,0,0.06); margin-bottom: 45px; }
p { color: #7f8c8d; font-size: 14px; margin-top: -5px; margin-bottom: 20px; }
</style>
</head>
<body>
<div class="table-container">
<h2>Table 1: Performance Relative to Master Default (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master default (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.09%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.80%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.30%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.56%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.91%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.77%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.28%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.34%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.64%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.06%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.80%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.78%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.87%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.24%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.16%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.59%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.40%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.53%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.20%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.16%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.06%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">77.29%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.68%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">97.11%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">102.78%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">97.76%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">95.93%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">75.08%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">71.27%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">86.54%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">112.56%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">110.55%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.79%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">103.61%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.50%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">94.33%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.38%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">118.67%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">122.60%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">109.97%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">108.72%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">107.00%</td>
</tr>
</table>
</div>
<div class="table-container">
<h2>Table 2: Performance Relative to Master Interleave (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master default</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.92%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.71%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.22%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.49%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">102.84%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.70%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.21%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.66%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.30%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.72%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.46%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.43%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.53%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.90%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.84%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.44%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.25%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.37%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.04%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.90%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">129.39%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">130.27%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">125.65%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">132.98%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">126.49%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">124.13%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">97.14%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">140.31%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">121.42%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">157.93%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">155.12%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">140.01%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">145.37%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">139.60%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">106.02%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">106.42%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">125.81%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">129.97%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">116.58%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">115.26%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">113.44%</td>
</tr>
</table>
</div>
</body>
</html>
[application/x-compressed-tar] numabenchhackersreview-2026-06-30.tgz (49.0K, ../../CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com/3-numabenchhackersreview-2026-06-30.tgz)
download
[text/html] performance_report_run3.html (9.1K, ../../CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com/4-performance_report_run3.html)
download | inline:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Performance Evaluation Matrix</title>
<style>
body { font-family: 'Segoe UI', Arial, sans-serif; margin: 40px; color: #333; background-color: #fafafa; }
h2 { color: #2c3e50; border-bottom: 2px solid #bdc3c7; padding-bottom: 10px; margin-top: 0; }
.table-container { background: #fff; padding: 25px; border-radius: 6px; box-shadow: 0 2px 4px rgba(0,0,0,0.06); margin-bottom: 45px; }
p { color: #7f8c8d; font-size: 14px; margin-top: -5px; margin-bottom: 20px; }
</style>
</head>
<body>
<div class="table-container">
<h2>Table 1: Performance Relative to Master Default (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master default (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">118.07%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">115.29%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.55%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.76%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">115.50%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.88%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">113.40%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.12%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.74%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">104.22%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">105.09%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">104.36%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">104.70%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">103.59%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.15%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.98%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.67%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.27%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.38%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.32%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.51%</td>
</tr>
</table>
</div>
<div class="table-container">
<h2>Table 2: Performance Relative to Master Interleave (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master default</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">84.69%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">97.64%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">83.46%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">87.03%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">97.82%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">83.74%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">96.04%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">97.93%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.61%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.06%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">102.91%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.20%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.53%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.45%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.85%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.84%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.53%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.12%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.23%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.17%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.36%</td>
</tr>
</table>
</div>
</body>
</html>
[text/x-patch] v20260630-0008-0001-clock-sweep-cached-CPU-NUMA-node-and-.patch (6.3K, ../../CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com/5-v20260630-0008-0001-clock-sweep-cached-CPU-NUMA-node-and-.patch)
download | inline diff:
From 8513d188ed5ed999e72fc3a58046bbc1ff9f5688 Mon Sep 17 00:00:00 2001
From: Jakub Wartak <[email protected]>
Date: Tue, 30 Jun 2026 14:22:02 +0200
Subject: [PATCH v20260630-0008] clock-sweep: cached CPU/NUMA node and more
locality-aware balancing
Enhancements on top of 0001-0007, to have sligthly better NUMA locality
and perfromance.
1. Cache numa_node_of_cpu()/sched_getcpu() per backend in
ClockSweepPartitionIndex(), refreshing every CLOCKSWEEP_CPU_NODE_REFRESH
allocations rather than on every call (visible hot buffer path in perf)
2. CLOCKSWEEP_BALANCE_THRESHOLD - make it less likely to redirect on any
surplus of allocations (so scatter buffers LESS onto remote nodes).
With this, it redirects its allocations to other (remote?) partitions
when the allocation exceeds the per-partition average allocation rate
by this percentage factor .
3. Avoid redirects to "idle" partitions: a redirect partition target
must have some traffic which is at least 2x our demand. This elimnates
cold partitions, but we can still reach them using scan-all-partitions
fallback.
---
src/backend/storage/buffer/freelist.c | 85 +++++++++++++++++++++++----
1 file changed, 74 insertions(+), 11 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e677c71e0b3..d64c2c67eb6 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -55,6 +55,9 @@
*/
#define CLOCKSWEEP_HISTORY_COEFF 0.5
+/* How often backend should re-fetch the CPU/node on which it is running on? */
+#define CLOCKSWEEP_CPU_NODE_REFRESH 128
+
/*
* GUCs controlling the NUMA-aware clock-sweep behavior.
*
@@ -70,6 +73,7 @@
* clocksweep_scan_all_partitions - when enabled, looking for a free buffer
* scans all clock-sweep partitions (in a round-robin way), not just the
* backend's "home" partition.
+ *
*/
bool clocksweep_balance = true;
bool clocksweep_balance_recalc = true;
@@ -368,13 +372,29 @@ ClockSweepPartitionIndex(void)
#ifdef USE_LIBNUMA
if (shared_buffers_numa)
{
- int cpu;
+ /*
+ * Cache the CPU/NUMA node, refreshing only every CLOCKSWEEP_CPU_NODE_REFRESH
+ * allocations. It appears that sched_getcpu()/numa_node_of_cpu() are not free.
+ * On some platforms it take price of full system call, or the rest (x86_64?)
+ * is can be use VDSO optimization. The backend rarely migrates between NUMA
+ * nodes, and the balance logic only needs to notice migration after some time,
+ * so an occasional refresh is good enough.
+ */
+ static int cached_node = -1;
+ static uint32 refresh_counter = 0;
+
+ if (cached_node < 0 || (refresh_counter++ % CLOCKSWEEP_CPU_NODE_REFRESH) == 0)
+ {
+ int cpu;
- /* XXX do we need to check sched_getcpu is available, somehow? */
- if ((cpu = sched_getcpu()) < 0)
+ /* XXX do we need to check sched_getcpu is available, somehow? */
+ if ((cpu = sched_getcpu()) < 0)
elog(ERROR, "sched_getcpu failed: %m");
- node = numa_node_of_cpu(cpu);
+ /* XXX/JW: use libnuma wrapper for this */
+ cached_node = numa_node_of_cpu(cpu);
+ }
+ node = cached_node;
}
#endif
@@ -768,7 +788,8 @@ StrategySyncBalance(void)
uint32 total_allocs = 0, /* total number of allocations */
avg_allocs, /* average allocations (per partition) */
- delta_allocs = 0; /* sum of allocs above average */
+ delta_allocs = 0, /* sum of allocs above average */
+ redirect_cutoff; /* redirect only above this many allocs */
if (!clocksweep_balance || !clocksweep_balance_recalc)
return;
@@ -852,6 +873,20 @@ StrategySyncBalance(void)
return;
}
+ /*
+ * A partition only redirects allocations to other partitions when it
+ * exceeds the average by more than some threshold percent.
+ * Below this cutoff we keep allocations local, to preserve NUMA locality.
+ *
+ * TODO: maybe better value is possible. On 4s with 25 I've got good results,
+ * but with value of 50 I've got slight degradation. Maybe it should
+ * be equal to 100/numa_nodes ?
+ *
+ */
+#define CLOCKSWEEP_CUTOFF_THRESHOLD 25
+ redirect_cutoff = avg_allocs +
+ (uint32) ((uint64) avg_allocs * CLOCKSWEEP_CUTOFF_THRESHOLD / 100);
+
/*
* The actual rebalancing
*
@@ -884,10 +919,15 @@ StrategySyncBalance(void)
/* reset the weights to start from scratch */
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
- /* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ /*
+ * Does this partition exceed its fair share by more than the
+ * threshold? If not, keep all allocations local - redirecting them
+ * would push memory onto remote NUMA nodes for no real benefit when
+ * the load is already close to balanced.
+ */
+ if (allocs[i] <= redirect_cutoff)
{
- /* fewer - don't redirect any allocations elsewhere */
+ /* near fair share (or below) - keep allocations local */
balance[i] = 100;
}
else
@@ -902,22 +942,45 @@ StrategySyncBalance(void)
/* fraction of the "total" delta */
double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
- /* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ /* how much we keep local; we hand out the rest below */
+ int kept = 100;
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
/* How many allocations to receive from i-th partition? */
uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ int w;
+
+ /* do not redirect to ourselves */
+ if (j == i)
+ continue;
/* ignore partitions that don't need additional allocations */
if (allocs[j] > avg_allocs)
continue;
+ /*
+ * Only use other partitions that actually have demand of
+ * their own (avoid idle). If we fail, there's always the
+ * scan-all-partitions fallback.
+ *
+ * TODO:: just guessing,heuristics
+ */
+ if (allocs[j] < (avg_allocs / 2))
+ continue;
+
/* fraction to redirect */
- balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ w = (int) ((100.0 * receive_allocs / allocs[i]) + 0.5);
+ balance[j] = w;
+ kept -= w;
}
+
+ /* avoid negative balances */
+ if (kept > 0)
+ balance[i] = kept;
+ else
+ balance[i] = 1;
}
/* combine the old and new weights (hysteresis) */
--
2.43.0
[text/html] performance_report_run4.html (9.4K, ../../CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com/6-performance_report_run4.html)
download | inline:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Performance Evaluation Matrix</title>
<style>
body { font-family: 'Segoe UI', Arial, sans-serif; margin: 40px; color: #333; background-color: #fafafa; }
h2 { color: #2c3e50; border-bottom: 2px solid #bdc3c7; padding-bottom: 10px; margin-top: 0; }
.table-container { background: #fff; padding: 25px; border-radius: 6px; box-shadow: 0 2px 4px rgba(0,0,0,0.06); margin-bottom: 45px; }
p { color: #7f8c8d; font-size: 14px; margin-top: -5px; margin-bottom: 20px; }
</style>
</head>
<body>
<div class="table-container">
<h2>Table 1: Performance Relative to Master Default (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master default (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">91.38%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">93.38%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">98.57%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">94.32%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">81.72%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">84.25%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">76.65%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.98%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.79%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.75%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">102.13%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">87.25%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">87.75%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">87.32%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.79%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.87%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">103.10%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">103.76%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">90.77%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">92.42%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">92.02%</td>
</tr>
</table>
</div>
<div class="table-container">
<h2>Table 2: Performance Relative to Master Interleave (100% Baseline)</h2>
<p>Values within ±2% of the baseline are considered noise and are uncolored.</p>
<table border="1" style="border-collapse: collapse; text-align: center; font-family: Arial, sans-serif; width: 100%; border: 1px solid #ddd;">
<tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;">
<th style="padding: 12px 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 12px 10px; font-size: 13px;">Clients</th>
<th style="padding: 12px 10px; font-size: 12px;">master interleave (Baseline)</th>
<th style="padding: 12px 10px; font-size: 12px;">master default</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">optimized (numa=on, bal=on)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=off, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=off)</th>
<th style="padding: 12px 10px; font-size: 12px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">1</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">109.43%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.18%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">107.86%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">103.22%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">89.43%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">92.19%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">83.87%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.03%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.80%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.77%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.14%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">86.40%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">86.90%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">86.47%</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; font-size: 13px; background-color: #fafafa;">pgbenchS100krows</td>
<td style="padding: 10px 8px; font-weight: bold; font-size: 13px; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">100.00%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">99.22%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #7f8c8d;">101.07%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #28a745; font-weight: 500;">102.30%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #c3e6cb; color: #155724; font-weight: bold;">102.95%</td>
<td style="padding: 10px 8px; font-size: 13px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">90.06%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">91.69%</td>
<td style="padding: 10px 8px; font-size: 13px; color: #dc3545; font-weight: 500;">91.30%</td>
</tr>
</table>
</div>
</body>
</html>
^ permalink raw reply [nested|flat] 89+ messages in thread
* Re: Adding basic NUMA awareness
@ 2026-07-02 09:24 Jakub Wartak <[email protected]>
parent: Jakub Wartak <[email protected]>
0 siblings, 0 replies; 89+ messages in thread
From: Jakub Wartak @ 2026-07-02 09:24 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexey Makhmutov <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 30, 2026 at 2:51 PM Jakub Wartak
<[email protected]> wrote:
>
> On Mon, Jun 29, 2026 at 9:42 AM Jakub Wartak
> <[email protected]> wrote:
> >
> > On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <[email protected]> wrote:
> > >
> > > >> I have some results from a new round of benchmarks, and it's a bit
> > > >> disappointing. Or rather, there seem to be some issues that I can't
> > > >> figure out, causing regressions.
> > > > [..]
> > > >> This chart is for median latency (in milliseconds):
> > > >>
> > > >> clients master 0003 0004 0003/on 0004/on
> > > >> -------------------------------------------------------------
> > > >> 1 12767 12582 14509 12807 15307
> > > >> 8 14383 14355 14149 14069 16165
> > > >> 32 14756 15198 14836 14984 17128
> > > >> --------------------------------------------------------
> > > >> 1 103% 114% 100% 120%
> > > >> 8 101% 98% 98% 112%
> > > >> 32 102% 101% 102% 116%
> > > >>
>
> [..lots of variables..]
>
> > > I'll try, but if you could try running some experiments on your own,
> > > that might be helpful.
> > [..]
> > > > Hopefully next week I'll try to repro those numbers to see if I can
> > > > help more.
> > > >
> > >
> > > Thank you! That'd be great.
> >
> > Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
> > that fscachenuma proggie to aid us in troubleshooting.
> >
> > -J.
> >
> > [0] - https://github.com/jakubwartakEDB/fscachenuma
>
> Hi Tomas,
>
> OK, so I've run couple of tests and modified run.sh and also tried to fix
> some inefficiencies spotted while testing this. Note the attached
> performance matrix is in TPS (so more is better). Raw results/CSV and
> scripts are attached too.
>
> * run2 = 2 workloads, partitioned pgbench_accounts
> * run3 = just pgbenchS w/o partitioning + warmup
> * run4 = semi-like pgbenchS w/o partitioning but 100k rows + warmup
>
[..]
>
> Stil on the table:
>
> 1. maybe even better balancing is possible (?), but this one is seems enough?
> I'm out of other ideas, well other than the
> "shared-relation-use-by-foreign-node" idea described much earlier (but
> I won't be able to pull that off), so I'm not entering this rabbit hole
> any deeper.
See below, seems like not needed (?)
> 2. Digging into io_method=worker optimizations (answering question: are they
> necessary?) Maybe I'll throw in run5 quite soon, this is going to be
> crucial to answer.
OK, I'm attaching are results from mine runs 5 and 6:
- only seqconcurrscans was tested, well because for other workloads io_worker
method was not getting load for those workers (only seq scans were offloaded)
- checksums were disabled, because IMHO that would be unfair comparision
(AFAIR there are offloaded)
- those optimizations for 0008 "optimized (numa=on, bal=on)" easily beat
"patched (numa=on, bal=on)" and seem to be crucial. We get like 1.2x-1.4x
across every io_method, but only with 0008.
- even when then doing just those logical fully cached reads from fully VFS
cached case, io_urings shines (I've added raw TPS number to show this,
compare across tables e.g. io_uring vs sync 13.378/8.993=1.487x for
io_uring with NUMA, but for master's for io_uring:sync it was just 8.79/7.389
= 1.189x without NUMA); seems like io_uring is more lightweight to show
more benefits of remote memory latencies
- there's some more juice to get out of the balancer for 0-reuse workloads
(but IMHO it's pointless to squeeze more, it's hard already)
- I was probably wrong when expecting that io_worker's worker processes/queues
should get NUMA affinity. They don't need to be apparently for me to see
benefits (maybe they could be and it would even better, but meh).
So with ruling io_method impact (I speculated earlier that his could be it),
this means that you were either hitting lack of opimizations needed from
0008 or were impacted by lack of drop_caches before the runs
> Maybe You could run those run*.sh with master vs inst-patchset/optimized?
> (I'm not sure, maybe there's even different factor at play too...)
This is seems to be crucial now, to double confirm the results / loaded-tested
on your hw with 0008. (but that hardware really needs to have effective latency
difference between at least 2 NUMA nodes -- Intel's mlc is good for this);
maybe also tweak those 125% inside 0008 to some other values, I've got 4 nodes,
so 100/4=25%)
> 3. Potentially mentioned earlier BAS strategies (forcing just use of local
> partitions for known-to-be-only-local-users: CTAS/VACCUM/etc), but I'm
> afarid that's not for me as I would certainly break/violate some
> invisible to me boundary.
And this one is still potentially on the table as nice thing to have.
-J.
Attachments:
[text/html] performance_report_runs56.html (10.8K, ../../CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com/2-performance_report_runs56.html)
download | inline:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Extended IO-Method Metrics Dashboard</title>
<style>
body { font-family: 'Segoe UI', Arial, sans-serif; margin: 35px; color: #333; background-color: #f7f9fa; }
h1 { color: #2c3e50; font-size: 24px; border-bottom: 3px solid #34495e; padding-bottom: 10px; }
h2 { color: #2980b9; font-size: 17px; margin-top: 0; margin-bottom: 4px; }
p { color: #7f8c8d; font-size: 13px; margin-top: 0; margin-bottom: 15px; }
.matrix-box { background: #ffffff; padding: 20px; border-radius: 6px; box-shadow: 0 2px 5px rgba(0,0,0,0.04); margin-bottom: 35px; overflow-x: auto; }
</style>
</head>
<body>
<h1>Isolated Multi-Run Performance Dashboard</h1>
<div class="matrix-box">
<h2>Table View: Isolated IO_URING Architecture Comparisons</h2>
<p>Baseline anchor is configured to IO_URING Master Default. Variance within ±2% is stripped of color tags. Raw TPS values are provided below the percentage in brackets.</p>
<table border="1" style="border-collapse: collapse; text-align: center; width: 100%; font-size: 12px; border: 1px solid #ddd;">
<tr style="background-color: #f5f5f5; border-bottom: 2px solid #ddd;">
<th style="padding: 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 10px; font-size: 13px;">Clients</th>
<th style="padding: 10px; font-size: 11px;">master default (Baseline)</th>
<th style="padding: 10px; font-size: 11px;">master interleave</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=on)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[2.957]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">90.84%<br>[2.687]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">89.88% 🔴<br>[2.658]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">134.06% 🟢<br>[3.965]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">130.02%<br>[3.845]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">93.23%<br>[2.757]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">124.86%<br>[3.692]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">97.88%<br>[2.895]</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[8.790]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">103.50%<br>[9.097]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">85.30% 🔴<br>[7.497]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">152.20% 🟢<br>[13.378]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">140.32%<br>[12.333]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">95.82%<br>[8.422]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">133.25%<br>[11.712]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">121.57%<br>[10.686]</td>
</tr>
</table>
</div>
<div class="matrix-box">
<h2>Table View: Isolated SYNC Architecture Comparisons</h2>
<p>Baseline anchor is configured to SYNC Master Default. Variance within ±2% is stripped of color tags. Raw TPS values are provided below the percentage in brackets.</p>
<table border="1" style="border-collapse: collapse; text-align: center; width: 100%; font-size: 12px; border: 1px solid #ddd;">
<tr style="background-color: #f5f5f5; border-bottom: 2px solid #ddd;">
<th style="padding: 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 10px; font-size: 13px;">Clients</th>
<th style="padding: 10px; font-size: 11px;">master default (Baseline)</th>
<th style="padding: 10px; font-size: 11px;">master interleave</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=on)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[2.786]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">95.29%<br>[2.654]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">91.64% 🔴<br>[2.553]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">138.08% 🟢<br>[3.847]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">118.98%<br>[3.314]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">92.09%<br>[2.565]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">109.75%<br>[3.057]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">92.26%<br>[2.570]</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[7.389]</td>
<td style="padding: 10px 8px; color: #7f8c8d;">101.59%<br>[7.507]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">92.58% 🔴<br>[6.841]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">121.70%<br>[8.993]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">122.41% 🟢<br>[9.045]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">108.84%<br>[8.042]</td>
<td style="padding: 10px 8px; color: #7f8c8d;">101.55%<br>[7.504]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">102.45%<br>[7.570]</td>
</tr>
</table>
</div>
<div class="matrix-box">
<h2>Table View: Isolated WORKER Architecture Comparisons</h2>
<p>Baseline anchor is configured to WORKER Master Default. Variance within ±2% is stripped of color tags. Raw TPS values are provided below the percentage in brackets.</p>
<table border="1" style="border-collapse: collapse; text-align: center; width: 100%; font-size: 12px; border: 1px solid #ddd;">
<tr style="background-color: #f5f5f5; border-bottom: 2px solid #ddd;">
<th style="padding: 10px; font-size: 13px;">Benchmark</th>
<th style="padding: 10px; font-size: 13px;">Clients</th>
<th style="padding: 10px; font-size: 11px;">master default (Baseline)</th>
<th style="padding: 10px; font-size: 11px;">master interleave</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">optimized (numa=on, bal=on)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=off, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=off)</th>
<th style="padding: 10px; font-size: 11px;">patched (numa=on, bal=on)</th>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">8</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[2.638]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">102.16%<br>[2.695]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">94.72% 🔴<br>[2.499]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">141.19% 🟢<br>[3.725]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">125.12%<br>[3.301]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">96.20%<br>[2.538]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">120.36%<br>[3.175]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">102.14%<br>[2.695]</td>
</tr>
<tr style="border-bottom: 1px solid #eee;">
<td style="padding: 10px 8px; font-weight: bold; text-align: left; background-color: #fafafa;">seqconcurrscans</td>
<td style="padding: 10px 8px; font-weight: bold; background-color: #fafafa;">32</td>
<td style="padding: 10px 8px; color: #7f8c8d;">100.00%<br>[8.231]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">106.74%<br>[8.785]</td>
<td style="padding: 10px 8px; background-color: #f5c6cb; color: #721c24; font-weight: bold;">82.16% 🔴<br>[6.763]</td>
<td style="padding: 10px 8px; background-color: #c3e6cb; color: #155724; font-weight: bold;">138.05% 🟢<br>[11.363]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">128.48%<br>[10.575]</td>
<td style="padding: 10px 8px; color: #dc3545; font-weight: 500;">97.20%<br>[8.000]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">123.98%<br>[10.204]</td>
<td style="padding: 10px 8px; color: #28a745; font-weight: 500;">118.27%<br>[9.735]</td>
</tr>
</table>
</div>
</body>
</html>
[text/csv] runs_56.csv (14.9K, ../../CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com/3-runs_56.csv)
download | inline:
1 inst-master interleave seqconcurrscans 8 off off sync 2.676989 2878362 2715697 3015619 3066518 3075780
1 inst-master interleave seqconcurrscans 8 off off worker 2.654060 2876362 2738345 3022688 3088356 3095842
1 inst-master interleave seqconcurrscans 32 off off sync 7.547877 4120369 3768043 4211900 4349624 4486588
1 inst-master interleave seqconcurrscans 32 off off worker 8.903923 3574316 3291553 3755495 3908110 3994992
1 inst-master default seqconcurrscans 8 off off sync 2.755305 2775538 2655184 2942029 3017548 3052756
1 inst-master default seqconcurrscans 8 off off worker 2.583993 3013722 2819932 3112583 3228339 3234414
1 inst-master default seqconcurrscans 32 off off sync 7.391571 4133566 3886627 4270030 4424005 4540403
1 inst-master default seqconcurrscans 32 off off worker 8.487846 3608805 3217320 3997094 4355847 4503342
1 inst-patched default seqconcurrscans 8 off off sync 2.452780 3023552 2894398 3352546 3841396 3848795
1 inst-patched default seqconcurrscans 8 off off worker 2.551404 3000524 2851455 3182294 3346244 3352813
1 inst-patched default seqconcurrscans 8 on on sync 2.441181 3071738 2865927 3322115 3602335 3800045
1 inst-patched default seqconcurrscans 8 on on worker 2.659102 2965008 2713202 3103033 3266821 3368933
1 inst-patched default seqconcurrscans 8 on off sync 2.863182 2589652 2515474 2658561 2863917 3019368
1 inst-patched default seqconcurrscans 8 on off worker 3.101204 2825456 2345080 2912933 3080250 3166610
1 inst-patched default seqconcurrscans 32 off off sync 7.791841 3876079 3631905 4092114 4293015 4538026
1 inst-patched default seqconcurrscans 32 off off worker 8.285688 3761133 3427572 4078537 4574949 4852470
1 inst-patched default seqconcurrscans 32 on on sync 7.402330 4010668 3731710 4184473 4305269 4395753
1 inst-patched default seqconcurrscans 32 on on worker 9.635358 3200814 2997154 3332884 3568549 3602032
1 inst-patched default seqconcurrscans 32 on off sync 7.519016 4086156 3762435 4175968 4388826 4433452
1 inst-patched default seqconcurrscans 32 on off worker 10.573888 2975224 2721993 3242101 3467550 3672055
1 inst-optimized default seqconcurrscans 8 off off sync 2.506822 3112966 3070583 3186178 3295660 3319968
1 inst-optimized default seqconcurrscans 8 off off worker 2.485940 3164649 3152745 3204443 3305720 3309064
1 inst-optimized default seqconcurrscans 8 on on sync 3.350347 2302655 2071707 2464977 2586117 2739338
1 inst-optimized default seqconcurrscans 8 on on worker 3.263306 2315507 2214794 2426887 2600047 2738475
1 inst-optimized default seqconcurrscans 8 on off sync 3.890314 2031607 1881625 2166464 2225422 2237909
1 inst-optimized default seqconcurrscans 8 on off worker 3.900989 2117812 1884965 2394180 2424182 2435418
1 inst-optimized default seqconcurrscans 32 off off sync 6.955164 4507183 4285988 4672143 4836631 5085743
1 inst-optimized default seqconcurrscans 32 off off worker 6.357685 4881738 4453838 5111071 5341250 5481688
1 inst-optimized default seqconcurrscans 32 on on sync 8.952744 3477125 3301485 3597680 3689125 3804913
1 inst-optimized default seqconcurrscans 32 on on worker 10.696006 2810302 2577911 3141908 3724926 3914489
1 inst-optimized default seqconcurrscans 32 on off sync 9.089130 3311289 3156579 3565783 3803199 3906203
1 inst-optimized default seqconcurrscans 32 on off worker 11.456092 2648508 2375730 2940157 3419532 3660475
2 inst-master interleave seqconcurrscans 8 off off sync 2.649848 2840969 2700424 2992280 3249930 3279506
2 inst-master interleave seqconcurrscans 8 off off worker 2.702813 2828926 2678584 2882799 3062416 3074249
2 inst-master interleave seqconcurrscans 32 off off sync 7.685309 4079532 3632034 4201586 4323920 4426624
2 inst-master interleave seqconcurrscans 32 off off worker 8.835220 3511445 3129789 3709112 4060279 4216424
2 inst-master default seqconcurrscans 8 off off sync 2.789350 2773903 2514290 2908478 3192024 3238406
2 inst-master default seqconcurrscans 8 off off worker 2.647798 2813176 2754710 2962378 2999475 3020639
2 inst-master default seqconcurrscans 32 off off sync 7.340749 4174400 3961415 4288855 4406501 4490228
2 inst-master default seqconcurrscans 32 off off worker 7.723892 3971808 3619146 4343887 4632774 4876213
2 inst-patched default seqconcurrscans 8 off off sync 2.623109 2963358 2734704 3180449 3323782 3388275
2 inst-patched default seqconcurrscans 8 off off worker 2.517701 3170482 2921132 3287468 3355181 3365669
2 inst-patched default seqconcurrscans 8 on on sync 2.748338 2746574 2541907 2903318 3022644 3086268
2 inst-patched default seqconcurrscans 8 on on worker 2.717298 2909167 2741565 2962322 3105991 3157195
2 inst-patched default seqconcurrscans 8 on off sync 3.392044 2464543 2226467 2569815 2779765 2807772
2 inst-patched default seqconcurrscans 8 on off worker 3.161808 2522084 2450206 2726516 2890917 2930085
2 inst-patched default seqconcurrscans 32 off off sync 8.233592 3709123 3520725 3883269 4143046 4277373
2 inst-patched default seqconcurrscans 32 off off worker 7.541541 4153688 3696428 4496990 4978923 5135092
2 inst-patched default seqconcurrscans 32 on on sync 7.781163 3921041 3695276 4158926 4402412 4433617
2 inst-patched default seqconcurrscans 32 on on worker 9.635703 3206440 2975700 3424701 3717263 3806342
2 inst-patched default seqconcurrscans 32 on off sync 7.708562 3939991 3691300 4238350 4410194 4492357
2 inst-patched default seqconcurrscans 32 on off worker 9.876998 3044868 2840159 3336400 3945582 4087393
2 inst-optimized default seqconcurrscans 8 off off sync 2.625141 2892065 2778044 3080416 3259276 3261317
2 inst-optimized default seqconcurrscans 8 off off worker 2.468194 3201467 3150326 3279964 3329275 3360653
2 inst-optimized default seqconcurrscans 8 on on sync 3.344419 2298839 2247764 2366483 2424038 2476379
2 inst-optimized default seqconcurrscans 8 on on worker 3.347555 2298166 2240664 2492459 2562032 2593538
2 inst-optimized default seqconcurrscans 8 on off sync 4.035318 2063474 1978815 2111768 2154611 2175613
2 inst-optimized default seqconcurrscans 8 on off worker 3.748983 2129790 1931120 2355762 2392778 2425881
2 inst-optimized default seqconcurrscans 32 off off sync 6.891168 4505642 4203261 4661520 4924000 5134558
2 inst-optimized default seqconcurrscans 32 off off worker 6.934404 4464713 4185950 4784074 5003432 5378674
2 inst-optimized default seqconcurrscans 32 on on sync 8.797975 3343569 3275718 3555210 3726177 3767152
2 inst-optimized default seqconcurrscans 32 on on worker 10.505213 2861893 2582270 3151551 3560410 3983324
2 inst-optimized default seqconcurrscans 32 on off sync 8.916987 3382611 3234405 3549497 3755938 3820733
2 inst-optimized default seqconcurrscans 32 on off worker 11.483059 2664295 2453063 2896696 3235139 3460191
3 inst-master interleave seqconcurrscans 8 off off sync 2.636658 2864047 2746953 2961078 3122231 3213684
3 inst-master interleave seqconcurrscans 8 off off worker 2.728493 2862784 2750801 2955252 3017022 3072659
3 inst-master interleave seqconcurrscans 32 off off sync 7.286935 4040494 3823486 4312952 4530050 4659128
3 inst-master interleave seqconcurrscans 32 off off worker 8.616611 3505228 3265947 3761825 3949569 4084360
3 inst-master default seqconcurrscans 8 off off sync 2.812523 2724799 2612620 2854603 3025053 3041229
3 inst-master default seqconcurrscans 8 off off worker 2.682941 2911877 2736163 3083407 3149042 3158100
3 inst-master default seqconcurrscans 32 off off sync 7.435153 4163874 4018603 4381211 4583828 4649378
3 inst-master default seqconcurrscans 32 off off worker 8.480540 3620201 3320671 3949636 4306697 4482075
3 inst-patched default seqconcurrscans 8 off off sync 2.619982 2954689 2714683 3128327 3255473 3306191
3 inst-patched default seqconcurrscans 8 off off worker 2.544683 3087245 2903318 3245804 3307561 3326391
3 inst-patched default seqconcurrscans 8 on on sync 2.520698 2991886 2878950 3162206 3361788 3425765
3 inst-patched default seqconcurrscans 8 on on worker 2.707328 2846824 2658325 2955296 3058556 3134744
3 inst-patched default seqconcurrscans 8 on off sync 2.916950 2569341 2526521 2709409 2838303 2962610
3 inst-patched default seqconcurrscans 8 on off worker 3.263208 2655152 2213683 2758317 2822051 2847386
3 inst-patched default seqconcurrscans 32 off off sync 8.101158 3796130 3630512 3979367 4233115 4438484
3 inst-patched default seqconcurrscans 32 off off worker 8.173914 3702438 3374890 4079291 4375159 4924184
3 inst-patched default seqconcurrscans 32 on on sync 7.526843 3886131 3737504 4180397 4313255 4343782
3 inst-patched default seqconcurrscans 32 on on worker 9.932574 3120478 2950367 3246793 3408817 3540048
3 inst-patched default seqconcurrscans 32 on off sync 7.284427 4130936 3950051 4315347 4452422 4493960
3 inst-patched default seqconcurrscans 32 on off worker 10.162592 3035477 2806654 3216980 3502294 3701549
3 inst-optimized default seqconcurrscans 8 off off sync 2.526344 2964040 2719890 3186705 3597985 4044494
3 inst-optimized default seqconcurrscans 8 off off worker 2.542438 3079693 3063764 3152475 3199328 3214071
3 inst-optimized default seqconcurrscans 8 on on sync 3.248719 2295094 2220279 2383244 2516998 2525622
3 inst-optimized default seqconcurrscans 8 on on worker 3.292432 2312976 2227042 2409784 2450680 2452687
3 inst-optimized default seqconcurrscans 8 on off sync 3.614213 2189543 2160256 2213380 2241207 2358541
3 inst-optimized default seqconcurrscans 8 on off worker 3.525200 2222728 2088955 2362728 2499756 2627019
3 inst-optimized default seqconcurrscans 32 off off sync 6.676329 4552409 4335021 4827118 5021999 5087930
3 inst-optimized default seqconcurrscans 32 off off worker 6.996006 4271679 3907828 4657722 5337897 5550770
3 inst-optimized default seqconcurrscans 32 on on sync 9.384717 3268320 2984502 3463888 3700082 3738641
3 inst-optimized default seqconcurrscans 32 on on worker 10.523817 2914593 2658932 3182988 3494090 3700647
3 inst-optimized default seqconcurrscans 32 on off sync 8.971526 3400277 3151000 3561849 3707342 3831781
3 inst-optimized default seqconcurrscans 32 on off worker 11.148981 2713000 2469870 2992647 3308257 3732590
1 inst-master interleave seqconcurrscans 8 off off io_uring 2.757692 2835801 2182802 3201153 3320441 3504861
1 inst-master interleave seqconcurrscans 32 off off io_uring 9.172991 3451943 3092988 3611762 3757798 3790186
1 inst-master default seqconcurrscans 8 off off io_uring 2.802159 2800999 2058995 3160746 3369275 3438105
1 inst-master default seqconcurrscans 32 off off io_uring 8.828234 3505430 3206294 3677051 3862434 3920741
1 inst-patched default seqconcurrscans 8 off off io_uring 2.612943 2713449 2242707 3412828 3641257 3705966
1 inst-patched default seqconcurrscans 8 on on io_uring 2.841880 2436691 2267844 2921661 3224380 3392650
1 inst-patched default seqconcurrscans 8 on off io_uring 3.496911 2108477 1908023 2501377 2825366 3026078
1 inst-patched default seqconcurrscans 32 off off io_uring 8.398227 3705043 3370133 4001827 4325676 4568141
1 inst-patched default seqconcurrscans 32 on on io_uring 10.370220 2846213 2768619 2971775 3725478 3920787
1 inst-patched default seqconcurrscans 32 on off io_uring 11.654950 2667720 2530884 2754969 2904466 3109381
1 inst-optimized default seqconcurrscans 8 off off io_uring 2.667388 2530417 2300868 3356107 3667914 3743414
1 inst-optimized default seqconcurrscans 8 on on io_uring 4.105005 1713342 1570518 2269352 2471679 2568127
1 inst-optimized default seqconcurrscans 8 on off io_uring 3.997729 1861739 1612082 2290925 2460857 2512248
1 inst-optimized default seqconcurrscans 32 off off io_uring 7.574586 4087880 4006824 4263768 4477479 4572563
1 inst-optimized default seqconcurrscans 32 on on io_uring 12.286394 2447202 2288029 2633232 2979863 3225310
1 inst-optimized default seqconcurrscans 32 on off io_uring 13.447404 2286876 2037942 2468679 2842209 3016201
2 inst-master interleave seqconcurrscans 8 off off io_uring 2.578477 3068229 2810748 3160906 3364891 3419854
2 inst-master interleave seqconcurrscans 32 off off io_uring 8.968299 3442426 3110103 3597986 3748503 4032146
2 inst-master default seqconcurrscans 8 off off io_uring 3.009021 2240183 2070663 2957352 3340103 3470890
2 inst-master default seqconcurrscans 32 off off io_uring 8.787420 3594066 3226110 3819010 4029909 4145554
2 inst-patched default seqconcurrscans 8 off off io_uring 2.810223 2547611 2166569 3276789 3491404 3542468
2 inst-patched default seqconcurrscans 8 on on io_uring 2.963096 2519445 2276954 2866279 3006462 3088138
2 inst-patched default seqconcurrscans 8 on off io_uring 3.838555 2086107 1274256 2453471 2673574 2980587
2 inst-patched default seqconcurrscans 32 off off io_uring 8.362200 3799463 3338145 4111278 4236696 4346678
2 inst-patched default seqconcurrscans 32 on on io_uring 10.921405 2832916 2770838 2995430 3215413 3359570
2 inst-patched default seqconcurrscans 32 on off io_uring 11.582974 2649705 2566114 2768700 2870174 3003858
2 inst-optimized default seqconcurrscans 8 off off io_uring 2.440755 3331598 2642722 3427806 3547759 3563158
2 inst-optimized default seqconcurrscans 8 on on io_uring 3.752205 1940115 1722224 2361482 2568050 2688359
2 inst-optimized default seqconcurrscans 8 on off io_uring 3.757866 2213689 1637995 2320790 2376451 2481400
2 inst-optimized default seqconcurrscans 32 off off io_uring 7.343589 4068242 3990393 4187254 4443366 4517875
2 inst-optimized default seqconcurrscans 32 on on io_uring 12.548757 2431059 2278524 2602537 2852850 3071340
2 inst-optimized default seqconcurrscans 32 on off io_uring 13.869174 2208720 2067153 2377084 2627509 2829760
3 inst-master interleave seqconcurrscans 8 off off io_uring 2.723404 2890686 2347595 3132221 3248437 3300641
3 inst-master interleave seqconcurrscans 32 off off io_uring 9.150572 3446528 3104054 3624569 3864223 3950228
3 inst-master default seqconcurrscans 8 off off io_uring 3.060892 2297317 2072774 3043368 3265868 3329549
3 inst-master default seqconcurrscans 32 off off io_uring 8.753120 3547255 3269402 3771458 3945083 4011136
3 inst-patched default seqconcurrscans 8 off off io_uring 2.848215 2562533 2185589 3292310 3478793 3607794
3 inst-patched default seqconcurrscans 8 on on io_uring 2.878807 2520219 2223005 2866542 3403036 3509122
3 inst-patched default seqconcurrscans 8 on off io_uring 3.742019 2152091 1272418 2470810 2666517 2828932
3 inst-patched default seqconcurrscans 32 off off io_uring 8.506004 3637480 3299843 4089576 4236298 4484975
3 inst-patched default seqconcurrscans 32 on on io_uring 10.765485 2819704 2681353 2930211 3520548 3723343
3 inst-patched default seqconcurrscans 32 on off io_uring 11.898296 2583704 2492455 2678251 2819577 2880050
3 inst-optimized default seqconcurrscans 8 off off io_uring 2.866309 2454616 2320684 3187668 3482158 3924235
3 inst-optimized default seqconcurrscans 8 on on io_uring 3.678610 2200702 1664000 2383154 2494210 2541547
3 inst-optimized default seqconcurrscans 8 on off io_uring 4.138721 2068692 1581242 2255223 2378403 2473754
3 inst-optimized default seqconcurrscans 32 off off io_uring 7.573882 4211556 4082860 4411563 4568039 4658195
3 inst-optimized default seqconcurrscans 32 on on io_uring 12.164411 2511902 2379643 2623657 2816523 2991253
3 inst-optimized default seqconcurrscans 32 on off io_uring 12.817506 2349947 2183656 2510189 2770153 2944910
[application/x-shellscript] run6.sh (4.7K, ../../CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com/4-run6.sh)
download
[application/x-shellscript] run5.sh (4.7K, ../../CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com/5-run5.sh)
download
^ permalink raw reply [nested|flat] 89+ messages in thread
end of thread, other threads:[~2026-07-02 09:24 UTC | newest]
Thread overview: 89+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-07-01 19:07 Adding basic NUMA awareness Tomas Vondra <[email protected]>
2025-07-02 11:37 ` Ashutosh Bapat <[email protected]>
2025-07-02 12:36 ` Tomas Vondra <[email protected]>
2025-07-03 14:07 ` Ashutosh Bapat <[email protected]>
2025-07-09 16:35 ` Andres Freund <[email protected]>
2025-07-09 16:55 ` Greg Burd <[email protected]>
2025-07-09 17:23 ` Andres Freund <[email protected]>
2025-07-10 12:13 ` Burd, Greg <[email protected]>
2025-07-11 17:34 ` Burd, Greg <[email protected]>
2025-07-10 15:31 ` Tomas Vondra <[email protected]>
2025-07-11 16:06 ` Andres Freund <[email protected]>
2025-07-17 21:11 ` Tomas Vondra <[email protected]>
2025-07-18 16:46 ` Andres Freund <[email protected]>
2025-07-18 20:48 ` Tomas Vondra <[email protected]>
2025-07-18 21:03 ` Andres Freund <[email protected]>
2025-07-03 14:49 ` Dmitry Dolgov <[email protected]>
2025-07-04 11:05 ` Jakub Wartak <[email protected]>
2025-07-04 18:12 ` Tomas Vondra <[email protected]>
2025-07-07 12:31 ` Jakub Wartak <[email protected]>
2025-07-17 21:14 ` Tomas Vondra <[email protected]>
2025-07-25 10:27 ` Jakub Wartak <[email protected]>
2025-07-25 10:51 ` Tomas Vondra <[email protected]>
2025-07-28 14:19 ` Tomas Vondra <[email protected]>
2025-07-30 08:29 ` Jakub Wartak <[email protected]>
2025-07-30 11:10 ` Tomas Vondra <[email protected]>
2025-08-04 14:24 ` Tomas Vondra <[email protected]>
2025-08-07 09:24 ` Tomas Vondra <[email protected]>
2025-08-07 09:36 ` Tomas Vondra <[email protected]>
2025-08-09 00:25 ` Andres Freund <[email protected]>
2025-08-12 11:04 ` Tomas Vondra <[email protected]>
2025-08-12 14:24 ` Andres Freund <[email protected]>
2025-08-12 15:06 ` Tomas Vondra <[email protected]>
2025-08-13 15:16 ` Andres Freund <[email protected]>
2025-08-13 16:36 ` Tomas Vondra <[email protected]>
2025-09-11 08:32 ` Tomas Vondra <[email protected]>
2025-09-11 15:41 ` Tomas Vondra <[email protected]>
2025-09-18 21:04 ` Tomas Vondra <[email protected]>
2025-10-12 23:58 ` Alexey Makhmutov <[email protected]>
2025-10-13 11:09 ` Tomas Vondra <[email protected]>
2025-10-13 18:34 ` Alexey Makhmutov <[email protected]>
2025-10-15 17:02 ` Tomas Vondra <[email protected]>
2025-10-31 11:57 ` Tomas Vondra <[email protected]>
2025-11-04 12:10 ` Jakub Wartak <[email protected]>
2025-11-04 21:21 ` Tomas Vondra <[email protected]>
2025-11-06 14:02 ` Jakub Wartak <[email protected]>
2025-11-11 11:52 ` Tomas Vondra <[email protected]>
2025-11-17 09:23 ` Jakub Wartak <[email protected]>
2025-11-17 17:28 ` Tomas Vondra <[email protected]>
2025-11-21 18:49 ` Tomas Vondra <[email protected]>
2025-11-25 14:12 ` Jakub Wartak <[email protected]>
2025-11-26 16:19 ` Tomas Vondra <[email protected]>
2025-12-02 12:26 ` Jakub Wartak <[email protected]>
2025-12-08 20:02 ` Tomas Vondra <[email protected]>
2026-06-05 12:52 ` Tomas Vondra <[email protected]>
2026-06-16 08:16 ` Jakub Wartak <[email protected]>
2026-06-16 12:39 ` Tomas Vondra <[email protected]>
2026-06-17 12:13 ` Jakub Wartak <[email protected]>
2026-06-24 20:26 ` Tomas Vondra <[email protected]>
2026-06-25 12:19 ` Jakub Wartak <[email protected]>
2026-06-25 13:49 ` Tomas Vondra <[email protected]>
2026-06-29 07:42 ` Jakub Wartak <[email protected]>
2026-06-30 12:51 ` Jakub Wartak <[email protected]>
2026-07-02 09:24 ` Jakub Wartak <[email protected]>
2025-10-15 17:15 ` Tomas Vondra <[email protected]>
2025-07-08 03:04 ` Andres Freund <[email protected]>
2025-07-08 12:27 ` Tomas Vondra <[email protected]>
2025-07-08 12:56 ` Andres Freund <[email protected]>
2025-07-09 10:04 ` Jakub Wartak <[email protected]>
2025-07-09 17:13 ` Andres Freund <[email protected]>
2025-07-10 10:15 ` Jakub Wartak <[email protected]>
2025-07-05 07:09 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-07 12:01 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Tomas Vondra <[email protected]>
2025-07-07 14:51 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-07 22:35 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Tomas Vondra <[email protected]>
2025-07-08 01:47 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-08 02:14 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-07 23:25 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Andres Freund <[email protected]>
2025-07-08 01:55 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-08 12:34 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Tomas Vondra <[email protected]>
2025-07-08 16:06 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-08 21:26 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Tomas Vondra <[email protected]>
2025-07-09 06:40 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Cédric Villemain <[email protected]>
2025-07-09 08:09 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Bertrand Drouvot <[email protected]>
2025-07-10 15:20 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Tomas Vondra <[email protected]>
2025-07-09 17:57 ` Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach Andres Freund <[email protected]>
2025-07-09 19:42 ` Andres Freund <[email protected]>
2025-07-10 10:16 ` Jakub Wartak <[email protected]>
2025-07-10 14:17 ` Bertrand Drouvot <[email protected]>
2025-07-11 16:14 ` Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox