public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Jakub Wartak <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Adding basic NUMA awareness
Date: Thu, 7 Aug 2025 11:24:18 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAKZiRmy7VW5b0OKwHXtRSudCBFB48-kYnOEXeqrLfbTQ=5qQrA@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAKZiRmxemXXVAouzM4Ls7pA7e0u6CVSLJeL_phKkmGPOzvUv_g@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAKZiRmx3+GwaP3oiRVHavxCJh6KxhZVZp86kj50ZJAv51h2-gQ@mail.gmail.com>
<[email protected]>
<[email protected]>
Hi!
Here's a slightly improved version of the patch series.
The main improvement is related to rebalancing partitions of different
sizes (which can happen because the sizes have to be a multiple of some
minimal "chunk" determined by memory page size etc.). Part 0009 deals
with that by adjusting the allocations by partition size. It works OK,
but it's also true it matters less as the shared_buffers size increases
(as the relative difference between large/small partition gets smaller).
The other improvements are related to the pg_buffercache_partitions
view, showing the weights and (running) totals of allocations.
I plan to take a break from this patch series for a while, so this would
be a good time to take a look, do a review, run some tests etc. ;-)
One detail about the balancing I forgot to mention in my last message is
how the patch "distributes" allocations to match the balancing weights.
Consider for example the example weights from that message:
P1: [ 55, 45]
P2: [ 0, 100]
Imagine a backend located on P1 requests allocation of a buffer. The
weights say 55% buffers should be allocated from P1, and 45% should be
redirected to P2. One way to achieve that would be generating a random
number in [1, 100], and if it's [1,55] then P1, otherwise P2.
The patch does a much simpler thing - treat the weight as a "budget",
i.e. number of buffers to allocate before proceeding to the "next"
partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
and then goes back to P1 in a round-robin way. The advantage is it can
do away without a PRNG.
There's two things I'm not entirely sure about:
1) memory model - I'm not quite sure the current code ensures updates to
weights are properly "communicated" to the other processes. That is, if
the bgwriter recalculates the weights, will the other backends see the
new weights right away? Using a stale weights won't cause "failures",
the consequence is just a bit of imbalance. But it shouldn't stay like
that for too long, so maybe it'd be good to add some memory barriers or
something like that.
2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
assumes each core / piece of RAM is assigned to a particular NUMA node.
For the buffer partitioning the patch mostly cares about memory, as it
"locates" the buffers on different NUMA nodes. Which works mostly OK
(ignoring the issues with huge pages described in previous message).
But it also cares about the cores (and the node for each core), because
it uses that to pick the right partition for a backend. And here the
situation is less clear, because the CPUs don't need to be assigned to a
particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
$ numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3
node 0 size: 992 MB
node 0 free: 274 MB
node 1 cpus: 0 1 2 3
node 1 size: 1019 MB
node 1 free: 327 MB
node 2 cpus: 0 1 2 3
node 2 size: 1019 MB
node 2 free: 321 MB
node 3 cpus: 0 1 2 3
node 3 size: 955 MB
node 3 free: 251 MB
node 4 cpus: 0 1 2 3
node 4 size: 1019 MB
node 4 free: 332 MB
node 5 cpus: 0 1 2 3
node 5 size: 1019 MB
node 5 free: 342 MB
node 6 cpus: 0 1 2 3
node 6 size: 1019 MB
node 6 free: 352 MB
node 7 cpus: 0 1 2 3
node 7 size: 1014 MB
node 7 free: 339 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 10 10 10 10 10 10 10
1: 10 10 10 10 10 10 10 10
2: 10 10 10 10 10 10 10 10
3: 10 10 10 10 10 10 10 10
4: 10 10 10 10 10 10 10 10
5: 10 10 10 10 10 10 10 10
6: 10 10 10 10 10 10 10 10
7: 10 10 10 10 10 10 10 10
This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
are not assigned to particular nodes - each core is mapped to all 8 NUMA
nodes. I'm not sure what to do about this (or how getcpu() or libnuma
handle this). And can the situation be even more complicated?
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20250807-0011-NUMA-pin-backends-to-NUMA-nodes.patch (3.5K, ../[email protected]/2-v20250807-0011-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From 630e4f0c9ce995ff5a9f1aaf68e8c64836cadffb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v20250807 11/11] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 11259151a7d..dbb4cbb1bfa 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -766,6 +766,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ee4684d1b8..3f88659b49f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7b718760248..862341e137e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2156,6 +2156,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index cdeee8dccba..a97741c6707 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v20250807-0010-NUMA-interleave-PGPROC-entries.patch (46.9K, ../[email protected]/3-v20250807-0010-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From d2c7801011a394b5b9e00d60bf59e759b65e1bed Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v20250807 10/11] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 -
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 63 +-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 565 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 14 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 64 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 5acae31b836..ba54f69eeb4 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -38,3 +38,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 13014549d00..ee3aa8be2ce 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 15
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -104,6 +106,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -946,3 +949,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..087279a6a8e 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 5b65a855b29..fb52039e1a6 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -500,8 +500,6 @@ buffer_partitions_prepare(void)
if (numa_nodes < 1)
numa_nodes = 1;
- elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
-
/*
* XXX A bit weird. Do we need to worry about postmaster? Could this even
* run outside postmaster? I don't think so.
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index bbe29bc9729..878d1e33f61 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -510,7 +510,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index bf987aed8d3..3e86e4ca2ae 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1622,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1739,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2223,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2297,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2498,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2724,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2755,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2857,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3019,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3060,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3188,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3231,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3300,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3402,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3467,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3522,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3568,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3597,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3628,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3669,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3732,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3798,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 62f3471448e..c84a2a5f1bc 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..11259151a7d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -90,6 +98,31 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static Size get_memory_page_size(void); /* XXX duplicate with bufi_init.c */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +133,63 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -129,6 +214,25 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -140,6 +244,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -191,7 +298,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -205,6 +312,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -241,19 +350,115 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partitoion
+ * per NUMA node.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += (numa_nodes * sizeof(PGProcPartition));
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /*
+ * Now initialize the PGPROC partition registry with a single
+ * partition for all the procs.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += sizeof(PGProcPartition);
+
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +491,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,15 +639,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +705,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2296,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2371,222 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(LOG, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(LOG, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (numa_procs_interleave && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a11bc71a386..6ee4684d1b8 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0552ed62cc7..7b718760248 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2146,6 +2146,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 66baf2bf33e..cdeee8dccba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..d2d269941fc 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -443,8 +448,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8540d537a3e..ded2db30422 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1876,6 +1876,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.50.1
[text/x-patch] v20250807-0009-NUMA-weighted-clocksweep-balancing.patch (5.1K, ../[email protected]/4-v20250807-0009-NUMA-weighted-clocksweep-balancing.patch)
download | inline diff:
From 170673d4fe89bb9436c2af41216884fefa0a48ee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20250807 09/11] NUMA: weighted clocksweep balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index f2203cebcc8..bbe29bc9729 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -738,6 +738,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -764,16 +778,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -781,8 +806,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -845,6 +876,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -852,7 +887,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -866,22 +901,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.50.1
[text/x-patch] v20250807-0008-NUMA-clocksweep-allocation-balancing.patch (26.2K, ../[email protected]/5-v20250807-0008-NUMA-clocksweep-allocation-balancing.patch)
download | inline diff:
From f832fedb847004990e7e806691ab4338752a5db5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 31 Jul 2025 19:50:05 +0200
Subject: [PATCH v20250807 08/11] NUMA: clocksweep allocation balancing
If backends only allocate buffers from the "local" partition, this could
cause significant misbalance - some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 41 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 396 ++++++++++++++++--
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 5 +-
6 files changed, 420 insertions(+), 31 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 999bb2128f0..5acae31b836 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -27,7 +27,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 7ca075e6164..13014549d00 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 15
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -789,6 +791,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -826,6 +830,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 12, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 13, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 14, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 15, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -851,11 +861,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -866,7 +882,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
&buffers_free,
&complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -904,6 +928,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[11] = Int64GetDatum(buffer_allocs);
nulls[11] = false;
+ values[12] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[12] = false;
+
+ values[13] = Int64GetDatum(buffer_req_allocs);
+ nulls[13] = false;
+
+ values[14] = PointerGetDatum(array);
+ nulls[14] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7a8c45ac59c..97b6f973c26 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3623,6 +3623,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 17988b4fd53..f2203cebcc8 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -51,6 +51,22 @@ typedef struct BufferStrategyFreelist
uint64 consumed;
} BufferStrategyFreelist;
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 16
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -83,9 +99,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -153,7 +188,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -165,7 +226,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -294,32 +355,72 @@ calculate_partition_index()
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
+
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
- return &StrategyControl->sweeps[index];
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
* ChooseFreeList
- * Pick the buffer freelist to use, depending on the CPU and NUMA node.
- *
- * Without partitioned freelists (numa_partition_freelist=false), there's only
- * a single freelist, so use that.
- *
- * With partitioned freelists, we have multiple ways how to pick the freelist
- * for the backend:
- *
- * - one freelist per CPU, use the freelist for CPU the task executes on
- *
- * - one freelist per NUMA node, use the freelist for node task executes on
+ * pick a clocksweep partition based on NUMA node and CPU
*
- * - use fixed number of freelists, map processes to lists based on PID
+ * The number of freelist partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
*
- * There may be some other strategies, not sure. The important thing is this
- * needs to be refrecled during initialization, i.e. we need to create the
- * right number of lists.
+ * XXX Maybe this should use the same balancing strategy as clocksweep?
*/
static BufferStrategyFreelist *
ChooseFreeList(void)
@@ -417,7 +518,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -580,6 +681,224 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(LOG, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(LOG, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- prepare for sync of all partitions
*
@@ -606,6 +925,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -852,7 +1172,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1242,7 +1576,9 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
BufferStrategyFreelist *freelist;
ClockSweep *sweep;
@@ -1288,11 +1624,21 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
/* get the clocksweep stats too */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 907b160b4f7..38bd5511048 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,6 +449,7 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index a6795c5fee9..e1729f0ee14 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -353,7 +353,10 @@ extern void FreelistPartitionGetInfo(int idx,
uint32 *complete_passes,
uint32 *next_victim_buffer,
uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.50.1
[text/x-patch] v20250807-0007-NUMA-clockweep-partitioning.patch (40.1K, ../[email protected]/6-v20250807-0007-NUMA-clockweep-partitioning.patch)
download | inline diff:
From c2dbd991bf2720eb6d9295bffd61744f525a19c9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v20250807 07/11] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
---
.../pg_buffercache--1.6--1.7.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 476 ++++++++++--------
src/backend/storage/buffer/freelist.c | 238 +++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 6 +-
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 504 insertions(+), 261 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 95fd2d2a226..999bb2128f0 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -21,7 +21,13 @@ CREATE VIEW pg_buffercache_partitions AS
-- freelists
list_consumed bigint, -- buffers consumed from a freelist
list_remain bigint, -- buffers left in a freelist
- list_free bigint); -- number of free buffers
+ list_free bigint, -- number of free buffers
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 6d734464a22..7ca075e6164 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "list_free",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,6 +851,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -850,7 +864,9 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
- &buffers_free);
+ &buffers_free,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -876,6 +892,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffers_free);
nulls[7] = false;
+ values[8] = Int64GetDatum(complete_passes);
+ nulls[8] = false;
+
+ values[9] = Int32GetDatum(next_victim_buffer);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_total_allocs);
+ nulls[10] = false;
+
+ values[11] = Int64GetDatum(buffer_allocs);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index bd50535385f..7a8c45ac59c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3584,6 +3584,23 @@ BufferSync(int flags)
TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_scan);
}
+/*
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
+ *
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
+ *
+ * XXX might be better to have a per-partition struct with all the info
+ */
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
/*
* BgBufferSync -- Write out some dirty buffers in the pool.
*
@@ -3599,55 +3616,24 @@ bool
BgBufferSync(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
- int strategy_buf_id;
- uint32 strategy_passes;
uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
-
- /* Moving averages of allocation rate and clean-buffer density */
- static float smoothed_alloc = 0;
- static float smoothed_density = 10.0;
-
- /* Potentially these could be tunables, but for now, not */
- float smoothing_samples = 16;
- float scan_whole_pool_milliseconds = 120000.0;
-
- /* Used to compute how far we scan ahead */
- long strategy_delta;
- int bufs_to_lap;
- int bufs_ahead;
- float scans_per_alloc;
- int reusable_buffers_est;
- int upcoming_alloc_est;
- int min_scan_buffers;
-
- /* Variables for the scanning loop proper */
- int num_to_scan;
- int num_written;
- int reusable_buffers;
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
- /* Variables for final smoothed_density update */
- long new_strategy_delta;
- uint32 new_recent_alloc;
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
- /*
- * Find out where the freelist clock sweep currently is, and how many
- * buffer allocations have happened since our last call.
- */
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
/* Report buffer alloc counts to pgstat */
PendingBgWriterStats.buf_alloc += recent_alloc;
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -3660,223 +3646,285 @@ BgBufferSync(WritebackContext *wb_context)
}
/*
- * Compute strategy_delta = how many buffers have been scanned by the
- * clock sweep since last time. If first time through, assume none. Then
- * see if we are still ahead of the clock sweep, and if so, how many
- * buffers we could scan before we'd catch up with it and "lap" it. Note:
- * weird-looking coding of xxx_passes comparisons are to avoid bogus
- * behavior when the passes counts wrap around.
- */
- if (saved_info_valid)
- {
- int32 passes_delta = strategy_passes - prev_strategy_passes;
-
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* info obtained from freelist.c */
+ int strategy_buf_id;
+ uint32 strategy_passes;
+
+ /* Moving averages of allocation rate and clean-buffer density */
+ static float smoothed_alloc = 0;
+ static float smoothed_density = 10.0;
+
+ /* Potentially these could be tunables, but for now, not */
+ float smoothing_samples = 16;
+ float scan_whole_pool_milliseconds = 120000.0;
+
+ /* Used to compute how far we scan ahead */
+ long strategy_delta;
+ int bufs_to_lap;
+ int bufs_ahead;
+ float scans_per_alloc;
+ int reusable_buffers_est;
+ int upcoming_alloc_est;
+ int min_scan_buffers;
+
+ /* Variables for the scanning loop proper */
+ int num_to_scan;
+ int num_written;
+ int reusable_buffers;
+
+ /* Variables for final smoothed_density update */
+ long new_strategy_delta;
+ uint32 new_recent_alloc;
+
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
- Assert(strategy_delta >= 0);
+ /*
+ * Find out where the freelist clock sweep currently is, and how many
+ * buffer allocations have happened since our last call.
+ */
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
- if ((int32) (next_passes - strategy_passes) > 0)
+ /*
+ * Compute strategy_delta = how many buffers have been scanned by the
+ * clock sweep since last time. If first time through, assume none.
+ * Then see if we are still ahead of the clock sweep, and if so, how
+ * many buffers we could scan before we'd catch up with it and "lap"
+ * it. Note: weird-looking coding of xxx_passes comparisons are to
+ * avoid bogus behavior when the passes counts wrap around.
+ */
+ if (saved_info_valid)
{
- /* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
+
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
+
+ Assert(strategy_delta >= 0);
+
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
+ {
+ /* we're one pass ahead of the strategy point */
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
#endif
- }
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
- {
- /* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ }
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
+ {
+ /* on same pass, but ahead or at least not behind */
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
+#ifdef BGW_DEBUG
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
+#endif
+ }
+ else
+ {
+ /*
+ * We're behind, so skip forward to the strategy point and
+ * start cleaning from there.
+ */
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta);
#endif
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
+ }
}
else
{
/*
- * We're behind, so skip forward to the strategy point and start
- * cleaning from there.
+ * Initializing at startup or after LRU scanning had been off.
+ * Always start at the strategy point.
*/
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta);
+ elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
+ strategy_passes, strategy_buf_id);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ strategy_delta = 0;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
- }
- else
- {
- /*
- * Initializing at startup or after LRU scanning had been off. Always
- * start at the strategy point.
- */
-#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
- strategy_passes, strategy_buf_id);
-#endif
- strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
- }
- /* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ /* Update saved info for next time */
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* FIXME has to happen after all partitions */
+ /* saved_info_valid = true; */
- /*
- * Compute how many buffers had to be scanned for each new allocation, ie,
- * 1/density of reusable buffers, and track a moving average of that.
- *
- * If the strategy point didn't move, we don't update the density estimate
- */
- if (strategy_delta > 0 && recent_alloc > 0)
- {
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
- }
+ /*
+ * Compute how many buffers had to be scanned for each new allocation,
+ * ie, 1/density of reusable buffers, and track a moving average of
+ * that.
+ *
+ * If the strategy point didn't move, we don't update the density
+ * estimate
+ */
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
+ {
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
+ }
- /*
- * Estimate how many reusable buffers there are between the current
- * strategy point and where we've scanned ahead to, based on the smoothed
- * density estimate.
- */
- bufs_ahead = NBuffers - bufs_to_lap;
- reusable_buffers_est = (float) bufs_ahead / smoothed_density;
+ /*
+ * Estimate how many reusable buffers there are between the current
+ * strategy point and where we've scanned ahead to, based on the
+ * smoothed density estimate.
+ */
+ bufs_ahead = num_buffers - bufs_to_lap;
+ reusable_buffers_est = (float) bufs_ahead / smoothed_density;
- /*
- * Track a moving average of recent buffer allocations. Here, rather than
- * a true average we want a fast-attack, slow-decline behavior: we
- * immediately follow any increase.
- */
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
- else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
- smoothing_samples;
+ /*
+ * Track a moving average of recent buffer allocations. Here, rather
+ * than a true average we want a fast-attack, slow-decline behavior:
+ * we immediately follow any increase.
+ */
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
+ else
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
+ smoothing_samples;
- /* Scale the estimate by a GUC to allow more aggressive tuning. */
- upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
+ /* Scale the estimate by a GUC to allow more aggressive tuning. */
+ upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
- /*
- * If recent_alloc remains at zero for many cycles, smoothed_alloc will
- * eventually underflow to zero, and the underflows produce annoying
- * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
- * zero, there's no point in tracking smaller and smaller values of
- * smoothed_alloc, so just reset it to exactly zero to avoid this
- * syndrome. It will pop back up as soon as recent_alloc increases.
- */
- if (upcoming_alloc_est == 0)
- smoothed_alloc = 0;
+ /*
+ * If recent_alloc remains at zero for many cycles, smoothed_alloc
+ * will eventually underflow to zero, and the underflows produce
+ * annoying kernel warnings on some platforms. Once
+ * upcoming_alloc_est has gone to zero, there's no point in tracking
+ * smaller and smaller values of smoothed_alloc, so just reset it to
+ * exactly zero to avoid this syndrome. It will pop back up as soon
+ * as recent_alloc increases.
+ */
+ if (upcoming_alloc_est == 0)
+ smoothed_alloc = 0;
- /*
- * Even in cases where there's been little or no buffer allocation
- * activity, we want to make a small amount of progress through the buffer
- * cache so that as many reusable buffers as possible are clean after an
- * idle period.
- *
- * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
- * the BGW will be called during the scan_whole_pool time; slice the
- * buffer pool into that many sections.
- */
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ /*
+ * Even in cases where there's been little or no buffer allocation
+ * activity, we want to make a small amount of progress through the
+ * buffer cache so that as many reusable buffers as possible are clean
+ * after an idle period.
+ *
+ * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many
+ * times the BGW will be called during the scan_whole_pool time; slice
+ * the buffer pool into that many sections.
+ */
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
- if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
- {
+ if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
+ {
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
- upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
+ elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
+ upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
#endif
- upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
- }
-
- /*
- * Now write out dirty reusable buffers, working forward from the
- * next_to_clean point, until we have lapped the strategy scan, or cleaned
- * enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.
- */
+ upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
+ }
- num_to_scan = bufs_to_lap;
- num_written = 0;
- reusable_buffers = reusable_buffers_est;
+ /*
+ * Now write out dirty reusable buffers, working forward from the
+ * next_to_clean point, until we have lapped the strategy scan, or
+ * cleaned enough buffers to match our estimate of the next cycle's
+ * allocation requirements, or hit the bgwriter_lru_maxpages limit.
+ */
- /* Execute the LRU scan */
- while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
- {
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ num_to_scan = bufs_to_lap;
+ num_written = 0;
+ reusable_buffers = reusable_buffers_est;
- if (++next_to_clean >= NBuffers)
+ /* Execute the LRU scan */
+ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- next_to_clean = 0;
- next_passes++;
- }
- num_to_scan--;
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
+ wb_context);
- if (sync_state & BUF_WRITTEN)
- {
- reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- PendingBgWriterStats.maxwritten_clean++;
- break;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
+ }
+ num_to_scan--;
+
+ if (sync_state & BUF_WRITTEN)
+ {
+ reusable_buffers++;
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
+ {
+ PendingBgWriterStats.maxwritten_clean++;
+ break;
+ }
}
+ else if (sync_state & BUF_REUSABLE)
+ reusable_buffers++;
}
- else if (sync_state & BUF_REUSABLE)
- reusable_buffers++;
- }
- PendingBgWriterStats.buf_written_clean += num_written;
+ PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
- elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
- smoothed_density, reusable_buffers_est, upcoming_alloc_est,
- bufs_to_lap - num_to_scan,
- num_written,
- reusable_buffers - reusable_buffers_est);
+ elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
+ smoothed_density, reusable_buffers_est, upcoming_alloc_est,
+ bufs_to_lap - num_to_scan,
+ num_written,
+ reusable_buffers - reusable_buffers_est);
#endif
- /*
- * Consider the above scan as being like a new allocation scan.
- * Characterize its density and update the smoothed one based on it. This
- * effectively halves the moving average period in cases where both the
- * strategy and the background writer are doing some useful scanning,
- * which is helpful because a long memory isn't as desirable on the
- * density estimates.
- */
- new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
- {
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
+ /*
+ * Consider the above scan as being like a new allocation scan.
+ * Characterize its density and update the smoothed one based on it.
+ * This effectively halves the moving average period in cases where
+ * both the strategy and the background writer are doing some useful
+ * scanning, which is helpful because a long memory isn't as desirable
+ * on the density estimates.
+ */
+ new_strategy_delta = bufs_to_lap - num_to_scan;
+ new_recent_alloc = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ {
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
- scans_per_alloc, smoothed_density);
+ elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
+ new_recent_alloc, new_strategy_delta,
+ scans_per_alloc, smoothed_density);
#endif
+ }
+
+ /* hibernate if all partitions can hibernate */
+ hibernate &= (bufs_to_lap == 0 && recent_alloc_partition == 0);
}
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 5a63dad7f2c..17988b4fd53 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -52,17 +52,27 @@ typedef struct BufferStrategyFreelist
} BufferStrategyFreelist;
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* Clock sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -73,6 +83,19 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
@@ -88,6 +111,9 @@ typedef struct
int num_partitions;
int num_partitions_per_node;
+ /* clocksweep partitions */
+ ClockSweep *sweeps;
+
BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
@@ -127,6 +153,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -138,6 +165,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -145,14 +173,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -178,19 +206,23 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
}
static int
@@ -247,6 +279,28 @@ calculate_partition_index()
return index;
}
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
+}
+
/*
* ChooseFreeList
* Pick the buffer freelist to use, depending on the CPU and NUMA node.
@@ -363,7 +417,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -434,13 +488,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Nothing on the freelist, so run the "clock sweep" algorithm
*
- * XXX Should we also make this NUMA-aware, to only access buffers from
- * the same NUMA node? That'd probably mean we need to make the clock
- * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
- * subset of buffers. But that also means each process could "sweep" only
- * a fraction of buffers, even if the other buffers are better candidates
- * for eviction. Would that also mean we'd have multiple bgwriters, one
- * for each node, or would one bgwriter handle all of that?
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
*/
trycounter = NBuffers;
for (;;)
@@ -522,6 +580,46 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncStart -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -529,37 +627,44 @@ StrategyFreeBuffer(BufferDesc *buf)
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -647,6 +752,10 @@ StrategyShmemSize(void)
size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
num_partitions)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -665,6 +774,7 @@ StrategyInitialize(bool init)
int num_nodes;
int num_partitions;
int num_partitions_per_node;
+ char *ptr;
/* */
BufferPartitionParams(&num_partitions, &num_nodes);
@@ -692,7 +802,8 @@ StrategyInitialize(bool init)
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
- MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -707,12 +818,42 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* have to point the sweeps array to right after the freelists */
+ ptr = (char *) StrategyControl +
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions);
+ StrategyControl->sweeps = (ClockSweep *) ptr;
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -760,7 +901,6 @@ StrategyInitialize(bool init)
buf->freeNext = freelist->firstFreeBuffer;
freelist->firstFreeBuffer = i;
}
-
}
}
else
@@ -1100,9 +1240,12 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
}
void
-FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
{
BufferStrategyFreelist *freelist;
+ ClockSweep *sweep;
int cur;
/* stats */
@@ -1112,6 +1255,7 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
freelist = &StrategyControl->freelists[idx];
+ sweep = &StrategyControl->sweeps[idx];
/* stat */
SpinLockAcquire(&freelist->freelist_lock);
@@ -1141,4 +1285,14 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
*remain = cnt_remain;
*actually_free = cnt_free;
+
+ /* get the clocksweep stats too */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9dfbecb9fe4..907b160b4f7 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,7 +449,9 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index df127274190..a6795c5fee9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -349,7 +349,11 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void FreelistPartitionGetInfo(int idx,
uint64 *consumed, uint64 *remain,
- uint64 *actually_free);
+ uint64 *actually_free,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 03ca3b7c8bc..8540d537a3e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -427,6 +427,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.50.1
[text/x-patch] v20250807-0006-NUMA-partition-buffer-freelist.patch (20.5K, ../[email protected]/7-v20250807-0006-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From a505c4e23d60d5aac911c24391444ba1b8320bf0 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v20250807 06/11] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
This uses the buffer partitioning scheme introduced by the earlier
patch, i.e. the partitions will "align" with NUMA nodes, etc.
It also extends the "pg_buffercache_partitions" view, to include
information about each freelist (number of consumedd buffers, ...).
When allocating a buffer, it's taken from the correct freelist (same
NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
.../pg_buffercache--1.6--1.7.sql | 7 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/freelist.c | 349 ++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 4 +-
7 files changed, 366 insertions(+), 30 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index fb9003c011e..95fd2d2a226 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -16,7 +16,12 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- freelists
+ list_consumed bigint, -- buffers consumed from a freelist
+ list_remain bigint, -- buffers left in a freelist
+ list_free bigint); -- number of free buffers
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8baa7c7b543..6d734464a22 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -812,6 +812,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "list_consumed",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "list_remain",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "list_free",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -833,12 +839,19 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffers_consumed,
+ buffers_remain,
+ buffers_free;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
+ &buffers_free);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -854,6 +867,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(buffers_consumed);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffers_remain);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffers_free);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..5a63dad7f2c 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,41 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
/*
* The shared freelist control information.
@@ -39,8 +66,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,8 +76,19 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -157,6 +193,88 @@ ClockSweepTick(void)
return victim;
}
+static int
+calculate_partition_index()
+{
+ int rc;
+ unsigned cpu;
+ unsigned node;
+ int index;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->freelists[index];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +286,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +314,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +381,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +431,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -356,7 +492,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ /*
+ * We don't want to call ChooseFreeList() again, because we might get a
+ * completely different freelist - either a different partition in the
+ * same group, or even a different group if the NUMA node changed. But we
+ * can calculate the proper freelist from the buffer id.
+ */
+ int index = (BufferGetNode(buf->buf_id) * StrategyControl->num_partitions_per_node)
+ + (buf->buf_id % StrategyControl->num_partitions_per_node);
+
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[index];
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +515,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +583,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int p = 0; p < StrategyControl->num_partitions; p++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[p];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "NUMA: freelist partition %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ p,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -445,12 +632,20 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+ int num_nodes;
+
+ BufferPartitionParams(&num_partitions, &num_nodes);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /* size of freelist partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ num_partitions)));
return size;
}
@@ -467,6 +662,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* */
+ BufferPartitionParams(&num_partitions, &num_nodes);
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -484,7 +691,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
&found);
if (!found)
@@ -494,13 +702,10 @@ StrategyInitialize(bool init)
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +716,52 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < num_partitions; nfreelist++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(nfreelist, &node,
+ &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * Walk through buffers for each partition, add them to the list.
+ * Walk from the end, because we're adding the buffers to the
+ * beginning.
+ */
+
+ for (int i = last_buffer; i >= first_buffer; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+
+ /* add to the freelist */
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
+
+ }
}
else
Assert(!init);
@@ -847,3 +1098,47 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+{
+ BufferStrategyFreelist *freelist;
+ int cur;
+
+ /* stats */
+ uint64 cnt_remain = 0;
+ uint64 cnt_free = 0;
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[idx];
+
+ /* stat */
+ SpinLockAcquire(&freelist->freelist_lock);
+
+ *consumed = freelist->consumed;
+
+ cur = freelist->firstFreeBuffer;
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ cnt_remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ cnt_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ SpinLockRelease(&freelist->freelist_lock);
+
+ *remain = cnt_remain;
+ *actually_free = cnt_free;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..a11bc71a386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+bool numa_partition_freelist = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a21f20800fb..0552ed62cc7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2136,6 +2136,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..66baf2bf33e 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index deaf4f19fa4..df127274190 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -347,7 +347,9 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
-
+extern void FreelistPartitionGetInfo(int idx,
+ uint64 *consumed, uint64 *remain,
+ uint64 *actually_free);
/* inline functions */
--
2.50.1
[text/x-patch] v20250807-0005-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../[email protected]/8-v20250807-0005-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From e0fa771531b92dc096fd5a1580ded3224524fa65 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v20250807 05/11] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.50.1
[text/x-patch] v20250807-0004-NUMA-localalloc.patch (3.7K, ../[email protected]/9-v20250807-0004-NUMA-localalloc.patch)
download | inline diff:
From 7fec3b33652e92903a20276536477df33be722c8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v20250807 04/11] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 17 +++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 29 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 65d8cbfaed5..d986a1d18cf 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,19 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9570087aa60..a21f20800fb 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v20250807-0003-NUMA-interleaving-buffers.patch (38.4K, ../[email protected]/10-v20250807-0003-NUMA-interleaving-buffers.patch)
download | inline diff:
From bbe6427b4e0e871bcb7b2cc4ce11ad8aba62799c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 28 Jul 2025 14:01:37 +0200
Subject: [PATCH v20250807 03/11] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to 'off'), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by numa_buffers_interleave GUC (default: false)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
contrib/pg_buffercache/Makefile | 2 +-
.../pg_buffercache--1.6--1.7.sql | 26 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 626 +++++++++++++++++-
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 2 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 15 +
src/tools/pgindent/typedefs.list | 2 +
11 files changed, 775 insertions(+), 11 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..fb9003c011e
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,26 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae0291e6e96..8baa7c7b543 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -771,3 +773,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..5b65a855b29 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +32,19 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsArray;
+
+static Size get_memory_page_size(void);
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
@@ -70,19 +91,89 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +203,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize the registry of buffer partitions, and also move the
+ * memory to different NUMA nodes (if enabled by GUC)
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +241,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +254,68 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -184,5 +330,467 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+
+#define MIN_BUFFER_PARTITIONS 4
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(LOG, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (numa_partitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ part->numa_node = n;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(LOG, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (!numa_buffers_interleave || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ *num_partitions = numa_partitions;
+ *num_nodes = numa_nodes;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9570087aa60 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 52a71b138f7..9dfbecb9fe4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -323,6 +323,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -491,4 +492,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..deaf4f19fa4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -143,6 +143,20 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+typedef struct BufferPartition
+{
+ int numa_node;
+ int num_buffers;
+ int first_buffer;
+ int last_buffer;
+} BufferPartition;
+
+typedef struct BufferPartitions
+{
+ int npartitions;
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* forward declared, to avoid having to expose buf_internals.h here */
struct WritebackContext;
@@ -319,6 +333,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e6f2e93b2d6..03ca3b7c8bc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.50.1
[text/x-patch] v20250807-0002-nbtree-Use-ReadRecentBuffer-in_bt_getroot.patch (7.3K, ../[email protected]/11-v20250807-0002-nbtree-Use-ReadRecentBuffer-in_bt_getroot.patch)
download | inline diff:
From 730467cfcf1d1b7f0a22f61bd48a37371f80cce9 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 27 Jun 2023 14:07:34 -0700
Subject: [PATCH v20250807 02/11] nbtree: Use ReadRecentBuffer()
in_bt_getroot()
We tend to access the btree root page over and over, when descending the
index. It's quite expensive. Not really specific to NUMA, but it hurts
there much more.
Thomas Munro worked on this.
Discussion: https://www.postgresql.org/message-id/20230627020546.t6z4tntmj7wmjrfh%40awork3.anarazel.de
Discussion: https://www.postgresql.org/message-id/CA%2BhUKGJ8N_DRSB0YioinWjS2ycMpmOLy32mbBqVVztwBvXgyJA%40mail.gmail.com
---
src/backend/access/nbtree/nbtpage.c | 21 ++++++++++++++--
src/backend/access/nbtree/nbtsearch.c | 17 +++++++++----
src/backend/storage/buffer/bufmgr.c | 35 +++++----------------------
src/include/utils/backend_status.h | 4 +++
src/include/utils/rel.h | 4 +++
5 files changed, 45 insertions(+), 36 deletions(-)
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index c79dd38ee18..3cc26490f06 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -35,6 +35,7 @@
#include "storage/procarray.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/snapmgr.h"
static BTMetaPageData *_bt_getmeta(Relation rel, Buffer metabuf);
@@ -344,7 +345,7 @@ Buffer
_bt_getroot(Relation rel, Relation heaprel, int access)
{
Buffer metabuf;
- Buffer rootbuf;
+ Buffer rootbuf = InvalidBuffer;
Page rootpage;
BTPageOpaque rootopaque;
BlockNumber rootblkno;
@@ -373,7 +374,20 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
Assert(rootblkno != P_NONE);
rootlevel = metad->btm_fastlevel;
- rootbuf = _bt_getbuf(rel, rootblkno, BT_READ);
+
+ if (BufferIsValid(rel->rd_recent_root))
+ {
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, rootblkno,
+ rel->rd_recent_root))
+ {
+ rootbuf = rel->rd_recent_root;
+ _bt_lockbuf(rel, rootbuf, BT_READ);
+ _bt_checkpage(rel, rootbuf);
+ }
+ }
+
+ if (rootbuf == InvalidBuffer)
+ rootbuf = _bt_getbuf(rel, rootblkno, BT_READ);
rootpage = BufferGetPage(rootbuf);
rootopaque = BTPageGetOpaque(rootpage);
@@ -390,6 +404,7 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
P_RIGHTMOST(rootopaque))
{
/* OK, accept cached page as the root */
+ rel->rd_recent_root = rootbuf;
return rootbuf;
}
_bt_relbuf(rel, rootbuf);
@@ -555,6 +570,8 @@ _bt_getroot(Relation rel, Relation heaprel, int access)
rootopaque->btpo_level, rootlevel);
}
+ rel->rd_recent_root = rootbuf;
+
/*
* By here, we have a pin and read lock on the root page, and no lock set
* on the metadata page. Return the root page's buffer.
diff --git a/src/backend/access/nbtree/nbtsearch.c b/src/backend/access/nbtree/nbtsearch.c
index d69798795b4..e1e73b4aaf3 100644
--- a/src/backend/access/nbtree/nbtsearch.c
+++ b/src/backend/access/nbtree/nbtsearch.c
@@ -168,11 +168,17 @@ _bt_search(Relation rel, Relation heaprel, BTScanInsert key, Buffer *bufP,
* stack entry for this page/level. If caller ends up splitting a
* page one level down, it usually ends up inserting a new pivot
* tuple/downlink immediately after the location recorded here.
+ *
+ * FIXME: Unfortunately this isn't a usable gating condition, as
+ * vacuum uses BT_READ and needs the stack.
*/
- new_stack = (BTStack) palloc(sizeof(BTStackData));
- new_stack->bts_blkno = BufferGetBlockNumber(*bufP);
- new_stack->bts_offset = offnum;
- new_stack->bts_parent = stack_in;
+ if (false && access == BT_WRITE)
+ {
+ new_stack = (BTStack) palloc(sizeof(BTStackData));
+ new_stack->bts_blkno = BufferGetBlockNumber(*bufP);
+ new_stack->bts_offset = offnum;
+ new_stack->bts_parent = stack_in;
+ }
/*
* Page level 1 is lowest non-leaf page level prior to leaves. So, if
@@ -186,7 +192,8 @@ _bt_search(Relation rel, Relation heaprel, BTScanInsert key, Buffer *bufP,
*bufP = _bt_relandgetbuf(rel, *bufP, child, page_access);
/* okay, all set to move down a level */
- stack_in = new_stack;
+ if (false && access == BT_WRITE)
+ stack_in = new_stack;
}
/*
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 67431208e7f..bd50535385f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -685,7 +685,6 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
BufferDesc *bufHdr;
BufferTag tag;
uint32 buf_state;
- bool have_private_ref;
Assert(BufferIsValid(recent_buffer));
@@ -713,38 +712,16 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
else
{
bufHdr = GetBufferDescriptor(recent_buffer - 1);
- have_private_ref = GetPrivateRefCount(recent_buffer) > 0;
- /*
- * Do we already have this buffer pinned with a private reference? If
- * so, it must be valid and it is safe to check the tag without
- * locking. If not, we have to lock the header first and then check.
- */
- if (have_private_ref)
- buf_state = pg_atomic_read_u32(&bufHdr->state);
- else
- buf_state = LockBufHdr(bufHdr);
-
- if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag))
+ if (!PinBuffer(bufHdr, NULL) ||
+ !BufferTagsEqual(&tag, &bufHdr->tag))
{
- /*
- * It's now safe to pin the buffer. We can't pin first and ask
- * questions later, because it might confuse code paths like
- * InvalidateBuffer() if we pinned a random non-matching buffer.
- */
- if (have_private_ref)
- PinBuffer(bufHdr, NULL); /* bump pin count */
- else
- PinBuffer_Locked(bufHdr); /* pin for first time */
-
- pgBufferUsage.shared_blks_hit++;
-
- return true;
+ UnpinBuffer(bufHdr);
+ return false;
}
- /* If we locked the header above, now unlock. */
- if (!have_private_ref)
- UnlockBufHdr(bufHdr, buf_state);
+ pgBufferUsage.shared_blks_hit++;
+ return true;
}
return false;
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 3016501ac05..443c492cf34 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -97,6 +97,10 @@ typedef struct PgBackendGSSStatus
*/
typedef struct PgBackendStatus
{
+#ifdef pg_attribute_aligned
+ pg_attribute_aligned(PG_CACHE_LINE_SIZE)
+#endif
+
/*
* To avoid locking overhead, we use the following protocol: a backend
* increments st_changecount before modifying its entry, and again after
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b552359915f..7ac2fb1512f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -25,6 +25,7 @@
#include "rewrite/prs2lock.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
+#include "storage/buf.h"
#include "storage/smgr.h"
#include "utils/relcache.h"
#include "utils/reltrigger.h"
@@ -159,6 +160,9 @@ typedef struct RelationData
/* data managed by RelationGetIndexAttrBitmap: */
bool rd_attrsvalid; /* are bitmaps of attrs valid? */
+
+ Buffer rd_recent_root;
+
Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */
Bitmapset *rd_pkattr; /* cols included in primary key */
Bitmapset *rd_idattr; /* included in replica identity index */
--
2.50.1
[text/x-patch] v20250807-0001-allow-pgbench-to-pin-backends-threads.patch (11.6K, ../[email protected]/12-v20250807-0001-allow-pgbench-to-pin-backends-threads.patch)
download | inline diff:
From 9097f04069ab604146606a7f6eabe9a2ea78dca0 Mon Sep 17 00:00:00 2001
From: Ubuntu
<azureuser@hb176.osdv23jxzvxutpvofmy4uk5hcd.bx.internal.cloudapp.net>
Date: Sat, 31 May 2025 00:45:47 +0000
Subject: [PATCH v20250807 01/11] allow pgbench to pin backends/threads
Supported policies are "none" (default), "colocated" (both processes on
the same random core) and "random" (both processes on different cores).
---
src/bin/pgbench/pgbench.c | 348 ++++++++++++++++++++++++++++++++++++++
1 file changed, 348 insertions(+)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 125f3c7bbbe..95479861177 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -41,6 +41,7 @@
#include <time.h>
#include <sys/time.h>
#include <sys/resource.h> /* for getrlimit */
+#include <sys/sysinfo.h>
/* For testing, PGBENCH_USE_SELECT can be defined to force use of that code */
#if defined(HAVE_PPOLL) && !defined(PGBENCH_USE_SELECT)
@@ -300,6 +301,12 @@ static const char *dbName = NULL;
static char *logfile_prefix = NULL;
static const char *progname;
+#define CPU_PINNING_NONE 0
+#define CPU_PINNING_RANDOM 1
+#define CPU_PINNING_COLOCATED 2
+
+static int pinning_mode = CPU_PINNING_NONE;
+
#define WSEP '@' /* weight separator */
static volatile sig_atomic_t timer_exceeded = false; /* flag from signal
@@ -639,6 +646,8 @@ typedef struct
int64 cnt; /* client transaction count, for -t; skipped
* and failed transactions are also counted
* here */
+
+ int cpu; /* CPU to pin to, -1 = no pinning */
} CState;
/*
@@ -672,6 +681,8 @@ typedef struct
StatsData stats;
int64 latency_late; /* count executed but late transactions */
+
+ int cpu; /* CPU to pin to, -1 = no pinning */
} TState;
/*
@@ -841,6 +852,45 @@ static void add_socket_to_set(socket_set *sa, int fd, int idx);
static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
+/*
+ * random number generator for assigning CPUs to processes/threads
+ *
+ * This is not a simple generator, because we want to keep the outcome
+ * balanced - it's not just about the total number of threads/processes
+ * assigned to each core, because a thread is likely much lighter than a
+ * backend. So we want to distribute each of those uniformly.
+ *
+ * For example, assume we have 2 cores, and want to pin 2 threads and 2
+ * backends. We could pin 2 threads to core 0 and 2 backends to core 1.
+ * But that would not be balanced, because core 1 will have to execute
+ * much more work. Instead, we want to assign 1+1 to each core.
+ *
+ * Another restriction is that we don't want to assign both sides of
+ * a connection to the same core.
+ *
+ * All of that however applies to the 'random' mode. In the 'colocated'
+ * mode, we want to assign both sides of a connection to the same core.
+ * But we still pick the core randomly.
+ */
+typedef struct cpu_generator_state
+{
+ int ncpus; /* number of CPUs available */
+ int nitems; /* number of items in the queue */
+ int *nthreads; /* number of threads for each CPU */
+ int *nclients; /* number of processes for each CPU */
+ int *items; /* queue of CPUs to pick from */
+} cpu_generator_state;
+
+static cpu_generator_state cpu_generator_init(int ncpus);
+static void cpu_generator_refill(cpu_generator_state * state);
+static void cpu_generator_reset(cpu_generator_state * state);
+static int cpu_generator_thread(cpu_generator_state * state);
+static int cpu_generator_client(cpu_generator_state * state, int thread_cpu);
+static void cpu_generator_print(cpu_generator_state * state);
+static bool cpu_generator_check(cpu_generator_state * state);
+
+static void reset_pinning(TState *threads, int nthreads);
+
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
@@ -959,6 +1009,7 @@ usage(void)
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
" --show-script=NAME show builtin script code, then exit\n"
" --verbose-errors print messages of all errors\n"
+ " --pin-cpus MODE pin threads and backends to CPU (random or colocated)\n"
"\nCommon options:\n"
" --debug print debugging output\n"
" -d, --dbname=DBNAME database name to connect to\n"
@@ -6718,6 +6769,7 @@ main(int argc, char **argv)
{"verbose-errors", no_argument, NULL, 15},
{"exit-on-abort", no_argument, NULL, 16},
{"debug", no_argument, NULL, 17},
+ {"pin-cpus", required_argument, NULL, 18},
{NULL, 0, NULL, 0}
};
@@ -7071,6 +7123,15 @@ main(int argc, char **argv)
case 17: /* debug */
pg_logging_increase_verbosity();
break;
+ case 18: /* pin CPUs */
+ if (pg_strcasecmp(optarg, "random") == 0)
+ pinning_mode = CPU_PINNING_RANDOM;
+ else if (pg_strcasecmp(optarg, "colocated") == 0)
+ pinning_mode = CPU_PINNING_COLOCATED;
+ else
+ pg_fatal("invalid pinning method, expecting \"random\" or \"colocated\", got: \"%s\"",
+ optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -7368,6 +7429,96 @@ main(int argc, char **argv)
nclients_dealt += thread->nstate;
}
+ /* reset the CPU assignment, i.e. no pinning by default */
+ reset_pinning(threads, nthreads);
+
+ /* try to assign threads/clients to CPUs */
+ if (pinning_mode != CPU_PINNING_NONE)
+ {
+ int nprocs = get_nprocs();
+ cpu_generator_state state = cpu_generator_init(nprocs);
+
+retry:
+ /* start from scratch */
+ cpu_generator_reset(&state);
+
+ /* assign CPU to all threads */
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ thread->cpu = cpu_generator_thread(&state);
+ }
+
+ /*
+ * assign CPUs to backends, one at a time (for each thread)
+ *
+ * This helps to keep it balanced in the 'random' mode.
+ */
+ while (true)
+ {
+ /* did we find any unassigned backend? */
+ bool found = false;
+
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ /* skip backends with already assigned CPUs */
+ if (thread->state[j].cpu != -1)
+ continue;
+
+ if (pinning_mode == CPU_PINNING_RANDOM)
+ thread->state[j].cpu = cpu_generator_client(&state, thread->cpu);
+ else
+ {
+ state.nclients[thread->cpu]++;
+ thread->state[j].cpu = thread->cpu;
+ }
+
+ /* move to the next thread */
+ found = true;
+ break;
+ }
+ }
+
+ /* if we haven't found any unassigned backend, we're done */
+ if (!found)
+ break;
+ }
+
+ /* validate the assignments don't violate any of the restrictions */
+ if ((pinning_mode == CPU_PINNING_RANDOM) &&
+ !cpu_generator_check(&state))
+ {
+ reset_pinning(threads, nthreads);
+ goto retry;
+ }
+
+ /* print info about CPU assignments */
+ printf("============================\n");
+
+ cpu_generator_print(&state);
+
+ printf("----------------------------\n");
+
+ for (i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ printf("thread %d CPU %d\n", i, thread->cpu);
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ printf(" client %d CPU %d\n", j, thread->state[j].cpu);
+ }
+ }
+
+ printf("============================\n");
+ }
+
/* all clients must be assigned to a thread */
Assert(nclients_dealt == nclients);
@@ -7519,6 +7670,39 @@ threadRun(void *arg)
}
}
+ /* pin the thread and backends for it's connections to the same CPU */
+ if (pinning_mode != CPU_PINNING_NONE)
+ {
+ cpu_set_t *cpusetp;
+ size_t cpusetsize;
+ size_t nprocs = get_nprocs();
+
+ cpusetp = CPU_ALLOC(nprocs);
+ cpusetsize = CPU_ALLOC_SIZE(nprocs);
+
+ CPU_ZERO_S(cpusetsize, cpusetp);
+ CPU_SET_S(thread->cpu, cpusetsize, cpusetp);
+
+ /* thread 0 dess not have an actual pthread */
+ if (thread->thread)
+ pthread_setaffinity_np(thread->thread, cpusetsize, cpusetp);
+ else
+ sched_setaffinity(getpid(), cpusetsize, cpusetp);
+
+ /* determine PID of the backend, pin it to the same CPU */
+ for (int i = 0; i < nstate; i++)
+ {
+ pid_t pid = PQbackendPID(state[i].con);
+
+ CPU_ZERO_S(cpusetsize, cpusetp);
+ CPU_SET_S(state[i].cpu, cpusetsize, cpusetp);
+
+ sched_setaffinity(pid, cpusetsize, cpusetp);
+ }
+
+ CPU_FREE(cpusetp);
+ }
+
/* GO */
THREAD_BARRIER_WAIT(&barrier);
@@ -7992,3 +8176,167 @@ socket_has_input(socket_set *sa, int fd, int idx)
}
#endif /* POLL_USING_SELECT */
+
+static cpu_generator_state
+cpu_generator_init(int ncpus)
+{
+ struct timeval tv;
+
+ cpu_generator_state state;
+
+ state.ncpus = ncpus;
+
+ state.nthreads = pg_malloc(sizeof(int) * ncpus);
+ state.nclients = pg_malloc(sizeof(int) * ncpus);
+
+ state.nitems = ncpus;
+ state.items = pg_malloc(sizeof(int) * ncpus);
+ for (int i = 0; i < ncpus; i++)
+ {
+ state.nthreads[i] = 0;
+ state.nclients[i] = 0;
+ state.items[i] = i;
+ }
+
+ gettimeofday(&tv, NULL);
+ srand48(tv.tv_usec);
+
+ return state;
+}
+
+static void
+cpu_generator_refill(cpu_generator_state * state)
+{
+ struct timeval tv;
+
+ state->items = pg_realloc(state->items,
+ (state->nitems + state->ncpus) * sizeof(int));
+
+ gettimeofday(&tv, NULL);
+
+ srand48(tv.tv_usec);
+
+ for (int i = 0; i < state->ncpus; i++)
+ state->items[state->nitems++] = i;
+}
+
+static void
+cpu_generator_reset(cpu_generator_state * state)
+{
+ state->nitems = 0;
+ cpu_generator_refill(state);
+
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ state->nthreads[i] = 0;
+ state->nclients[i] = 0;
+ }
+}
+
+static int
+cpu_generator_thread(cpu_generator_state * state)
+{
+ if (state->nitems == 0)
+ cpu_generator_refill(state);
+
+ while (true)
+ {
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
+
+ state->items[idx] = state->items[state->nitems - 1];
+ state->nitems--;
+
+ state->nthreads[cpu]++;
+
+ return cpu;
+ }
+}
+
+static int
+cpu_generator_client(cpu_generator_state * state, int thread_cpu)
+{
+ int min_clients;
+ bool has_valid_cpus = false;
+
+ for (int i = 0; i < state->nitems; i++)
+ {
+ if (state->items[i] != thread_cpu)
+ {
+ has_valid_cpus = true;
+ break;
+ }
+ }
+
+ if (!has_valid_cpus)
+ cpu_generator_refill(state);
+
+ min_clients = INT_MAX;
+ for (int i = 0; i < state->nitems; i++)
+ {
+ if (state->items[i] == thread_cpu)
+ continue;
+
+ min_clients = Min(min_clients, state->nclients[state->items[i]]);
+ }
+
+ while (true)
+ {
+ int idx = lrand48() % state->nitems;
+ int cpu = state->items[idx];
+
+ if (cpu == thread_cpu)
+ continue;
+
+ if (state->nclients[cpu] != min_clients)
+ continue;
+
+ state->items[idx] = state->items[state->nitems - 1];
+ state->nitems--;
+
+ state->nclients[cpu]++;
+
+ return cpu;
+ }
+}
+
+static void
+cpu_generator_print(cpu_generator_state * state)
+{
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ printf("CPU %d threads %d clients %d\n", i, state->nthreads[i], state->nclients[i]);
+ }
+}
+
+static bool
+cpu_generator_check(cpu_generator_state * state)
+{
+ int min_count = INT_MAX,
+ max_count = 0;
+
+ for (int i = 0; i < state->ncpus; i++)
+ {
+ min_count = Min(min_count, state->nthreads[i] + state->nclients[i]);
+ max_count = Max(max_count, state->nthreads[i] + state->nclients[i]);
+ }
+
+ return (max_count - min_count <= 1);
+}
+
+static void
+reset_pinning(TState *threads, int nthreads)
+{
+ /* reset the CPU assignment, i.e. no pinning by default */
+ for (int i = 0; i < nthreads; i++)
+ {
+ TState *thread = &threads[i];
+
+ thread->cpu = -1;
+
+ for (int j = 0; j < thread->nstate; j++)
+ {
+ thread->state[j].cpu = -1;
+ }
+ }
+}
--
2.50.1
view thread (89+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox