public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Andres Freund <[email protected]>
Cc: Jakub Wartak <[email protected]>
Cc: Alexey Makhmutov <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Adding basic NUMA awareness
Date: Fri, 5 Jun 2026 14:52:35 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com>
<[email protected]>
<CAKZiRmxwN+qMpbijCLPix_y6mwSjgus2CPPj=1+uFo9fQG-Knw@mail.gmail.com>
<[email protected]>
<uezi46xhhbvdjgdi6wl7iqgfcdh4jmnnyzbfovdcrck6ywqa7j@fj3yimxvekk6>
<[email protected]>
<zndz3dlmp7xlypczxjbwdfrey3masto6vuwpnzjvgunslprv25@rucfkdgfpb4p>
<rsjxzhnxxfq5i5yxv66mhinb42o3vzmqpkbfpexpkk5prreh2l@jyp73gsyyfzn>
<[email protected]>
<clx4zzd7kau4vvh5ynu5ssxg3jqfqzurgcbtotytzgzkhb3nis@qfl5xwv44yad>
<[email protected]>
Hi,
Here's an updated version of the NUMA patch series, based on some recent
discussions about this (some at pgconf.dev, but not only that),
The main change is I significantly simplified some of the parts. Whe
patch from 20251126 was ~190K, the new version is maybe 100K, so about
half. Some of that is thanks to dropping the PGPROC partitioning
entirely, but the remaining parts are smaller too. I realize it's not a
great metric, of course.
In this message I'll explain the changes since 20251126. I'm yet to do a
thorough performance evaluation and see if it helps, I'll post that in
the next couple days.
The current patch series has these parts:
v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch
-------------
Somewhat unrelated, I find this useful for benchmarking and as a
baseline (what would happen if we just interleaved the shared segment).
v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch
-------------
Just adds a small registry of partitions (ranges of shared buffers),
stored in shared memory, and pg_buffercache interface to inspect it.
Merely a foundation for the following patches.
v20260605-0003-NUMA-shared-buffers-partitioning.patch
-------------
The interesting part, that places some of the partitions to NUMA nodes.
v20260605-0004-clock-sweep-basic-partitioning.patch
v20260605-0005-clock-sweep-balancing-of-allocations.patch
v20260605-0006-clock-sweep-scan-all-partitions.patch
-------------
Patches that gradually partition clock-sweep. Ultimately, it should
probably be squashed into a single commit (each commit fixes some sort
of issue in the naive partitioning in 0004). But I kept them separate
because it's easier to review / understand what the issue is.
what changed?
-------------
First, I dropped the PGPROC partitioning. We may revisit that in the
future (not sure), but for now it was just a distraction and I see it as
less impactful than shared buffers / clock-sweep.
I also simplified the GUC to use a single on/off parameter (instead of
the debug_io_direct-like approach). We can revisit that, but for now
this seems more convenient.
The most significant change in the remaining parts is simplification of
the shared buffer partitioning. In particular, the partitioning is now
"best-effort" when it maps memory to shared buffers. Let me remind that
NUMA works at memory page granularity - we can't map arbitrary ranges of
memory to a node, it needs to be whole memory pages.
The 20251126 patch went into great lengths to (a) make sure BufferBlocks
and BufferDescriptors start at memory page boundary, are the partitions
are also properly aligned (both for blocks and descriptors). That was a
lot of code, it needs to happen even before we know if huge pages are
used, partitions might have been of (very) different sizes, and so on.
The new patch abandons this "perfect" partitioning, and instead does a
best-effort. It splits the buffers as evenly as possible, i.e. all
partitions have (NBuffers/npartitions) buffers, and then locates as much
memory as possible to a selected NUMA node.
With 4K pages, that's always the whole partition. With huge pages (which
is expected of relevant NUMA systems), there may be a couple buffers at
the beginning/end of a partition. But it's less than one memory page,
per partition, and we expect the systems to have 10s or 100s of GBs, so
in the bigger scheme of things it's negligible (fractions of a percent).
For buffer descriptors the math is a bit worse - descriptors need much
less memory, but even there it should not be more than ~1%.
Seems perfectly fine to me. Or rather, the extra complexity does not
seem worth the possible benefit.
This also allowed dropping a part of the "clock-sweep partitioning"
patches, dealing with cases when the partitions are of different sizes.
With this new best-effort scheme the difference is at most 1 buffer, and
we can just ignore that.
questions
---------
At this point, my main question is whether there's a better way to
partition clock-sweep and/or do the balancing of allocations between
partitions. I believe it does work, but I have a feeling there might be
a more elegant way to do this kind of stuff (like an established
balancing algorithm of some sort).
The other thing I need to verify is how this behaves with
kernel.nr_hugepages. With some earlier versions it was easy to end in a
situation where everything seemed to work, but then much later the
kernel realized it does not have enough huge pages on a particular NUMA
node and crashed with a segfault (or was it sigbus?).
Of course, the other question is performance validation - does it even
help? I plan to repeat the various experiments mentioned in this thread
(by Andres and others) on available NUMA machines. But if someone has an
idea for another benchmark (and/or what metric to measure, not just the
usual duration), let me know.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20260605-0006-clock-sweep-scan-all-partitions.patch (6.2K, ../[email protected]/2-v20260605-0006-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From 977831d85199b4b1b8d65de1131e4001459878fb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:39:38 +0200
Subject: [PATCH v20260605 6/6] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 83 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index a543fb12b21..1ac1e3e3490 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -184,6 +184,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint64 *buf_state);
/*
* clocksweep allocation balancing
@@ -251,10 +254,9 @@ static int clocksweep_partition_budget = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -486,7 +488,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -545,33 +548,61 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * a process "sweeps" only a fraction of buffers, even if the other buffers
- * are better candidates for eviction. Maybe there should be some logic to
- * "steal" buffers from other partitions or other nodes?
- *
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Needs to scan partitions if needed, as a fallback.
- *
- * XXX Would that also mean we should have multiple bgwriters, one for each
- * node, or would one bgwriter still handle all nodes?
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint64 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint64 old_buf_state;
uint64 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -599,7 +630,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -618,7 +649,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index f68e08e5697..ae977297849 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.54.0
[text/x-patch] v20260605-0005-clock-sweep-balancing-of-allocations.patch (27.4K, ../[email protected]/3-v20260605-0005-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From ad46b08a3eaa018b562fdd4ca786c4c23d358ba5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:28:10 +0200
Subject: [PATCH v20260605 5/6] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 428 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 471 insertions(+), 21 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 92176fed7f8..43d2e84f9d2 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -20,7 +20,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 739c63b0cfc..c91f2bc5b4a 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -31,7 +33,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -889,6 +891,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -920,6 +924,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -941,11 +951,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -954,8 +970,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -984,6 +1008,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[9] = false;
+
+ values[10] = Int64GetDatum(buffer_req_allocs);
+ nulls[10] = false;
+
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 02c75f82e5b..62e541abebd 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4135,6 +4135,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 2d56579682e..a543fb12b21 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -35,6 +35,26 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX We need to make ClockSweep fixed-size, so that we can have an array
+ * in shared memory. The easiest way is to pick a sufficiently high value
+ * that no system will actually need. 32 seems high enough.
+ *
+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average, so that we don't flap too much. The higher the value, the
+ * more the old value affects the result.
+ *
+ * XXX Doesn't this obscure the interpretation of weights as probabilities to
+ * allocate from a given partition? Does it still sum to 100%? I don't think
+ * so, it's just a fraction of allocations to go from a given partition.
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -68,9 +88,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that partition. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * Backends use the budget from it's "home" partition, so that a busy
+ * partitions (with a lot of processes on that NUMA node etc.) spread
+ * the allocations evenly.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -140,7 +183,66 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets a
+ * budget for allocating buffers from other partitions. A process that
+ * "exhausts" a budget in it's home partition gets redirected to the other
+ * partitions, driven by the budgets.
+ *
+ * For example, a partition may have budget [25, 25, 25, 25], which means
+ * each of the 4 partitions should get 1/4 of allocations. Or the buget
+ * can be [50, 50, 0, 0], which means all allocations will go to the first
+ * two partitions (one of them being the "home" one);
+ *
+ * We could do that based on a random number generator, but for now we
+ * simply treat the values as a budget, i.e. a number of allocations to
+ * serve from other partitions, and move in round-robin way.
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balanced. We can make the budgets higher
+ * (say, to match the expected number of allocations, i.e. bout the average
+ * number of allocations from the past interval). Or maybe configurable.
+ *
+ * XXX We should always start allocating from the "home" partition, i.e.
+ * from from it, and only then redirect to other partitions.
+ *
+ * XXX It probably is not great all the processes from that "home"
+ * partition are coordinated, and move to between partitions at about the
+ * same time. Not sure what to do about this.
+ *
+ * XXX We should also prefer other partitions from the same NUMA node (if
+ * there are some). Probably by setting the budgets.
+ *
+ * FIXME Explain at which point are the budgets recalculated, by which
+ * process, and how that affects other processes allocating buffers.
+ */
+
+/*
+ * The "optimal" clock-sweep partition. After a backend gets moved to a
+ * different NUMA node, we restart the balancing so that it uses the
+ * correct "budget" from the new home partition.
+ */
+static int clocksweep_partition_home = -1;
+
+/*
+ * The partition the backend is currently allocating from (either the
+ * home one, or one of the redirected ones).
+ */
+static int clocksweep_partition_current = -1;
+
+/*
+ * The number of buffers to allocate from the current partition.
+ */
+static int clocksweep_partition_budget = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -152,7 +254,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -300,11 +402,68 @@ ClockSweepPartitionIndex(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
+ /* What's the "optimal" partition for this backend? */
int index = ClockSweepPartitionIndex();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Was the process migrated to a different NUMA node? If the home partition
+ * changed, we need to reset the budget and start over, so that we correctly
+ * prefer "nearby" partitions etc.
+ *
+ * XXX Could this be a problem when processes move all the time? I don't
+ * think so - if a process moves between many partitions, that alone will
+ * spread the allocations over partitions. Similarly, if there are many
+ * processes, that should make it even more even.
+ */
+ if (clocksweep_partition_home != index)
+ {
+ clocksweep_partition_home = index;
+ clocksweep_partition_current = index;
+ clocksweep_partition_budget = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_home != -1);
+ Assert(clocksweep_partition_current != -1);
+ Assert(clocksweep_partition_budget >= 0);
+
+ /*
+ * When balancing allocations, redirect the allocations to other partitions
+ * according to the budgets. We move through partitions in a round-robin way,
+ * after allocating the "budget" of allocations from the current one.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of budget from the current partition? Move to the next one
+ * with non-zero budget.
+ */
+ while (clocksweep_partition_budget == 0)
+ {
+ /* wrap around at the end */
+ clocksweep_partition_current++;
+ if (clocksweep_partition_current >= StrategyControl->num_partitions)
+ clocksweep_partition_current = 0;
+
+ clocksweep_partition_budget
+ = sweep->balance[clocksweep_partition_current];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the current allocation */
+ --clocksweep_partition_budget;
+
+ /*
+ * Account for the allocation in the "home" partition, so that the next
+ * round of rebalancing (recalculating the budgets) knows about the
+ * allocation traffic in various partitions.
+ */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition_current];
}
/*
@@ -381,7 +540,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* between CPUs / NUMA nodes in between, these call may pick different
* partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -485,6 +644,229 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition budgets, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * This means an allocation may be requested in partition A (i.e. the
+ * home partition of the process requesting it), but end up allocating
+ * the buffer in partition B. We have a counter for both - the number of
+ * allocations requested in a partition, and the number of allocations
+ * actually handled by that partition. The former is used for calculating
+ * weights, the latter is used only for monitoring.
+ *
+ * The balancing happens in intervals - it adjusts future allocations
+ * based on stats about recent allocations, namely:
+ *
+ * - numBufferAllocs - number of allocations served by a partition
+ *
+ * - numRequestedAllocs - number of allocatios requested in a partition
+ *
+ * We're trying to smooth numBufferAllocs in the next interval, based on
+ * numRequestedAllocs measured in the last interval.
+ *
+ * The balancing algorithm works like this:
+ *
+ * - the target (average number of allocations per partition) is calculated
+ * from total number of allocations requested in the last intervaal
+ *
+ * - partitions get divided into two groups - those with more allocation
+ * requests than the target, and those with fewer requests
+ *
+ * - we "distribute" the delta (which is the same between the groups)
+ * between the groups (one has more, the other fewer)
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX Currently this does not give preference to other partitions on the
+ * same NUMA node (redirect to it first), but it could.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocation requests for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * XXX We lock the partitions one by one, so this is not a perfectly
+ * consistent snapshot of the counts, and the resets happen before we
+ * update the weights too. But we're only looking for heuristics, so
+ * this should be good enough.
+ *
+ * XXX A similar issue applies to the counter reset later - we haven't
+ * updated the weights yet, so some of the requests counted for the next
+ * interval will be redirected per current weights. Should be fine, it's
+ * just an approximate heuristics, and there should be very few requests in
+ * between. Alternatively, we could reset the request counters when setting
+ * the new weights, and just ignore the couple requests in between.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /* Calculate the "fair share" of allocations per partition. */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state for each partition, i.e. how
+ * many more/fewer allocations it handled relative to the average.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip rebalancing when there's not enough activity, and just keep the
+ * current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary. And maybe we should
+ * do the rabalancing in this case anyway, it's likely cheap and on a big
+ * system 10% can be quite a lot.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * The actual rebalancing
+ *
+ * Partition with fewer than average allocations, should not redirect any
+ * allocations to other partitions. So just use weights with a single
+ * non-zero weight for the partition itself.
+ *
+ * Partition with more than average allocations, should not receive any
+ * redirected allocations, and instead it should redirect excess allocations
+ * to other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of a
+ * partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from it.
+ *
+ * XXX We should add hysteresis, so that it does not oscillate or something
+ * like that. Maybe CLOCKSWEEP_HISTORY_COEFF already does that?
+ *
+ * XXX Ideally, the alternative partitions to use first would be the other
+ * partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -657,7 +1039,21 @@ StrategyCtlShmemInit(void *arg)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1025,8 +1421,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1034,11 +1432,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5ab0cee4281..887314d43f4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,6 +593,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e0bb4cc1df1..02833b19b0c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -411,11 +411,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.54.0
[text/x-patch] v20260605-0004-clock-sweep-basic-partitioning.patch (34.0K, ../[email protected]/4-v20260605-0004-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 5ab7aba5f8af84b748180eb6caa2b165aceb2590 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:16:55 +0200
Subject: [PATCH v20260605 4/6] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 202 +++++++----
src/backend/storage/buffer/freelist.c | 333 ++++++++++++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 486 insertions(+), 104 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index a6e49fd1652..92176fed7f8 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -14,7 +14,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index e3efeeda675..739c63b0cfc 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -912,6 +912,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -933,12 +941,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -954,6 +972,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);
+ nulls[5] = false;
+
+ values[6] = Int32GetDatum(next_victim_buffer);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_total_allocs);
+ nulls[7] = false;
+
+ values[8] = Int64GetDatum(buffer_allocs);
+ nulls[8] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index cc398db124d..02c75f82e5b 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3826,33 +3826,34 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
- *
- * This is called periodically by the background writer process.
+ * Information saved between calls so we can determine the strategy point's
+ * advance rate and avoid scanning already-cleaned buffers.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX Does it actually make sense to split all of this information per
+ * partition? For example, does per-partition advance rate mean anything?
+ * Maybe we should have a global advance rate? Although, if we want to
+ * keep enough clean buffers in each partition, maybe having per-partition
+ * rates makes sense.
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+typedef struct BufferSyncPartition
+{
+ int prev_strategy_buf_id;
+ uint32 prev_strategy_passes;
+ int next_to_clean;
+ uint32 next_passes;
+} BufferSyncPartition;
+
+static BufferSyncPartition *saved_info = NULL;
+static bool saved_info_valid = false;
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition,
+ BufferSyncPartition *saved)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3880,25 +3881,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3910,17 +3902,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - saved->prev_strategy_passes;
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - saved->prev_strategy_buf_id;
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (saved->next_passes - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - saved->next_to_clean;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3928,11 +3920,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (saved->next_passes == strategy_passes &&
+ saved->next_to_clean >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (saved->next_to_clean - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3952,9 +3944,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3968,15 +3960,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ saved->next_to_clean = strategy_buf_id;
+ saved->next_passes = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ saved->prev_strategy_buf_id = strategy_buf_id;
+ saved->prev_strategy_passes = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3984,9 +3977,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3996,7 +3989,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -4004,10 +3997,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -4034,7 +4027,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -4059,20 +4052,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(saved->next_to_clean, true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++saved->next_to_clean >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ saved->next_to_clean = first_buffer;
+ saved->next_passes++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -4086,7 +4079,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -4116,8 +4109,83 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ /* allocate space for per-partition information between calls */
+ if (saved_info == NULL)
+ {
+ /*
+ * XXX Not great it's using malloc(), but how else to allocate a
+ * variable-length array?
+ */
+ saved_info = malloc(sizeof(BufferSyncPartition) * num_partitions);
+ }
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition,
+ &saved_info[partition]);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 53ef5239e8d..2d56579682e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -36,17 +36,28 @@
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock sweep partition */
+ int32 node;
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -57,11 +68,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+
+ /* cached info about freelist partitioning */
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -108,6 +140,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -119,6 +152,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -126,14 +160,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -159,19 +193,118 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /*
+ * Make sure we've calculated a buffer in the range of the partition. Buffer
+ * IDs are 1-based, we're calculating 0-based indexes.
+ */
+ Assert((victim >= 0) && (victim < sweep->numBuffers));
+ Assert(BufferIsValid(1 + sweep->firstBuffer + victim));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * ClockSweepPartitionIndex
+ * pick the clock-sweep partition to use based on PID and NUMA node
+ *
+ * With libnuma, use the NUMA node and PID to pick the partition. Otherwise
+ * use just PID (as if there's a single NUMA node).
+ *
+ * XXX This should also check if buffers are NUMA-partitioned, not just if
+ * compiled with libnuma.
+ */
+static int
+ClockSweepPartitionIndex(void)
+{
+ int node = 0,
+ index;
+ pid_t pid = MyProcPid;;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * If buffers are NUMA-partitioned, determine the partition using the NUMA
+ * node and PID. Without NUMA assume everything is a single NUMA node 0, and
+ * we pick the partition based on PID.
+ */
+#ifdef USE_LIBNUMA
+ if (shared_buffers_numa)
+ {
+ int cpu;
+
+ /* XXX do we need to check sched_getcpu is available, somehow? */
+ if ((cpu = sched_getcpu()) < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+ }
+#endif
+
+ /*
+ * We should't get unexpected NUMA nodes, not considered when setting up the
+ * buffer partitions. It could happen if the allowed NUMA nodes get adjusted
+ * at runtime, but at this point we just create partitions for all existing
+ * nodes. We could plan for allowed partitions, but then what if those get
+ * disabled, and the user allows some other partitions?
+ */
+ if ((node < 0) || (node > StrategyControl->num_nodes))
+ elog(ERROR, "node out of range: %d > %u", node, StrategyControl->num_nodes);
+
+ /*
+ * Calculate the partition index. Nodes have the same number of partitions,
+ * and we use the PID to pick one of those (for a given node). If there's
+ * only a single partition per node, we can ignore PID and use node directly.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = node;
+ }
+ else
+ {
+ /* use PID to pick one of node's partitions */
+ index = (node * StrategyControl->num_partitions_per_node)
+ + (pid % StrategyControl->num_partitions_per_node);
+ }
+
+ /* should have a valid partition index */
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ return index;
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and PID
+ *
+ * Pick a partition mapped to the NUMA node the backend is currently running
+ * on, and use PID if there are multiple partitions per node. Without NUMA
+ * supported/enabled, use just PID.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = ClockSweepPartitionIndex();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -242,10 +375,37 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* We count buffer allocation requests so that the bgwriter can estimate
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
+ *
+ * XXX It's not quite right we call ChooseClockSweep twice - now, and then
+ * a couple lines later (through ClockSweepTick). If the process moves
+ * between CPUs / NUMA nodes in between, these call may pick different
+ * partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * a process "sweeps" only a fraction of buffers, even if the other buffers
+ * are better candidates for eviction. Maybe there should be some logic to
+ * "steal" buffers from other partitions or other nodes?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Needs to scan partitions if needed, as a fallback.
+ *
+ * XXX Would that also mean we should have multiple bgwriters, one for each
+ * node, or would one bgwriter still handle all nodes?
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -325,6 +485,48 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* XXX Do we need the lock, if we're only accessing atomics? Surely not. */
+ /* XXX Are we ever calling this without num_buf_alloc? */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -332,37 +534,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -394,8 +603,14 @@ StrategyNotifyBgWriter(int bgwprocno)
static void
StrategyCtlShmemRequest(void *arg)
{
+ int num_partitions;
+
+ /* get the number of buffer partitions */
+ BufferPartitionsCalculate(NULL, &num_partitions, NULL);
+
ShmemRequestStruct(.name = "Buffer Strategy Status",
- .size = sizeof(BufferStrategyControl),
+ .size = offsetof(BufferStrategyControl, sweeps) +
+ mul_size(num_partitions, sizeof(ClockSweep)),
.ptr = (void **) &StrategyControl
);
}
@@ -408,12 +623,42 @@ StrategyCtlShmemInit(void *arg)
{
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Remember the number of partitions */
+ BufferPartitionsParams(&StrategyControl->num_nodes,
+ &StrategyControl->num_partitions,
+ &StrategyControl->num_partitions_per_node);
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].node = node;
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -777,3 +1022,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e944cee2e91..5ab0cee4281 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -593,7 +593,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
/* buf_table.c */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 1cf09e8fb7c..e0bb4cc1df1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -410,6 +410,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index ae977297849..f68e08e5697 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ea756015249..183570b4d43 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -451,6 +451,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.54.0
[text/x-patch] v20260605-0003-NUMA-shared-buffers-partitioning.patch (26.6K, ../[email protected]/5-v20260605-0003-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 4672b0b53f68177ca76b9400e7f9ca4172a08ddb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 23:27:13 +0200
Subject: [PATCH v20260605 3/6] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc.
In cases like pre-warming a database from a single worker (e.g. using
pg_prewarm), we may end up with severely unbalanced memory distribution
(with most memory located on a single NUMA node). Unbalanced allocation
may put a lot of pressure on the memory system on a small number of NUMA
nodes, limiting the bandwidth etc.
With zone_reclaim, the kernel would eventually move some of the memory
to other nodes, but that tends to take a long time and is unpredictable.
This change forces even distribution of shared buffers on all NUMA
nodes, improving predictability, reducing the time needed for warmup
during benchmarking, etc. It's also less dependent on what the CPU
scheduler decides to do (which cores get used for the warmup.)
The effect is similar to
numactl --interleave=all
in that the buffers are distributed on the NUMA nodes evenly, but
there's also a number of important differences.
Firstly, it's applied only to shared buffers (and buffer descriptors),
not to the whole shared memory segment. It's possible to enable memory
interleaving using the shmem_interleave GUC, introduced in an earlier
patch in this series.
NUMA works at the granularity of a memory page, which is typically
either 4K or 2MB (hugepage), but other sizes are possible. For systems
where NUMA matters, we expect large amounts of memory (hundreds of
gigabytes) and hugepages enabled. But not necessarily.
The partitioning scheme is best-effort with respect to memory page size.
The shared buffers do not "align" with memory pages (i.e. a partition
may not end at the memory page boundary), in which case we simply locate
just the section of the partition with complete memory pages. This means
there may be ~one unmapped memory page between partitions. Considering
the expected amounts of memory, this is negligible, and the alternative
would be a significant amount of complexity to align the pages and
enforce "allowed" partition sizes.
Buffer descriptors are affected by this too, and the effect may be more
significant, simply because the descriptors are much smaller (~64B). So
the array is smaller, and a single 2MB memory page is worth ~32K buffer
descriptors. But with large systems it's still negligible.
The "buffer partitions" may not be 1:1 with NUMA nodes. We want to allow
clock-sweep partitioning even on non-NUMA systems, or when running only
on a small number of NUMA nodes. There's a minimal number of partitions
(default: 4), and a node may get multiple partitions. Nodes always get
the same number of partitions (e.g. with 3 NUMA nodes there will be 6
partitions in total, as each node gets 2 partitions).
The feature is enabled by dshared_buffers_numa GUC (default: false).
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/buf_init.c | 244 ++++++++++++++++--
src/backend/storage/buffer/freelist.c | 9 +
src/backend/utils/misc/guc_parameters.dat | 6 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/port/pg_numa.h | 7 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 8 +
src/port/pg_numa.c | 112 ++++++++
10 files changed, 392 insertions(+), 36 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index d62b8339bfc..a6e49fd1652 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer); -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index d678cb045ba..e3efeeda675 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -904,11 +904,13 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -926,28 +928,32 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index e593b02e0ca..1f93a31d451 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,12 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proclist.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -71,9 +79,12 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
-/* number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
+bool shared_buffers_numa = false;
/*
* Register shared memory area for the buffer pool.
@@ -81,6 +92,10 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
static void
BufferManagerShmemRequest(void *arg)
{
+ int nparts;
+
+ BufferPartitionsCalculate(NULL, &nparts, NULL);
+
ShmemRequestStruct(.name = "Buffer Descriptors",
.size = NBuffers * sizeof(BufferDescPadded),
/* Align descriptors to a cacheline boundary. */
@@ -103,7 +118,7 @@ BufferManagerShmemRequest(void *arg)
);
ShmemRequestStruct(.name = "Buffer Partition Registry",
- .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ .size = nparts * sizeof(BufferPartition),
/* Align descriptors to a cacheline boundary. */
.alignment = PG_CACHE_LINE_SIZE,
.ptr = (void **) &BufferPartitionsRegistry,
@@ -134,6 +149,10 @@ BufferManagerShmemInit(void *arg)
/*
* Initialize the buffer partition registry first, before other parts
* have a chance to touch the memory.
+ *
+ * Also moves memory to different NUMA nodes (if enabled by a GUC).
+ * Do this before the loop that initializes buffer headers etc. which
+ * may fault some of the memory pages etc.
*/
BufferPartitionsInit();
@@ -231,35 +250,203 @@ BufferPartitionsInit(void)
{
int buffer = 0;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
- int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+ int nnodes,
+ npartitions,
+ npartitions_per_node;
- BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ int buffers_per_partition,
+ buffers_remaining;
- for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
- {
- BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+ /* calculate partitioning parameters */
+ BufferPartitionsCalculate(&nnodes, &npartitions, &npartitions_per_node);
+
+ /* paranoia */
+ Assert(nnodes > 0);
+ Assert(npartitions >= MIN_BUFFER_PARTITIONS);
+ Assert((npartitions % nnodes) == 0);
+ Assert((npartitions_per_node * nnodes) == npartitions);
- int num_buffers = part_buffers;
- if (n < remaining_buffers)
- num_buffers += 1;
+ BufferPartitionsRegistry->nnodes = nnodes;
+ BufferPartitionsRegistry->npartitions = npartitions;
+ BufferPartitionsRegistry->npartitions_per_node = npartitions_per_node;
- remaining_buffers -= num_buffers;
+ /* regular partition size, the first couple get an extra buffer */
+ buffers_per_partition = (NBuffers / npartitions);
+ buffers_remaining = (NBuffers % buffers_per_partition);
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ /* should have all the buffers */
+ Assert((buffers_per_partition * npartitions + buffers_remaining) == NBuffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ /*
+ * Now walk the partitions, and set the buffer range. Optionally, place
+ * the partitions on a given node (for all partitions at once).
+ */
+ for (int n = 0; n < nnodes; n++)
+ {
+ for (int p = 0; p < npartitions_per_node; p++)
+ {
+ int idx = (n * npartitions_per_node) + p;
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ /*
+ * Assign to the NUMA node, but only with shared_buffers_numa=on.
+ *
+ * XXX we should get an actual node ID from the mask, in case the
+ * task is restricted to only some nodes.
+ */
+ part->numa_node = (shared_buffers_numa) ? n : -1;
+
+ /* The first couple partitions may get an extra buffer. */
+ part->num_buffers = buffers_per_partition;
+ if (idx < buffers_remaining)
+ part->num_buffers += 1;
+
+ /* remember the buffer range */
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (part->num_buffers - 1);
+
+ /* remember start of the next partition */
+ buffer += part->num_buffers;
+ }
- buffer += num_buffers;
+#ifdef USE_LIBNUMA
+ /*
+ * Now try to locate buffers and buffer descriptors to the node (all
+ * partitions for the node at once).
+ */
+ if (shared_buffers_numa)
+ {
+ Size numa_page_size = pg_numa_page_size();
+
+ int part_first,
+ part_last,
+ buff_first,
+ buff_last;
+
+ char *startptr,
+ *endptr;
+
+ /* first/last partition for this node */
+ part_first = (n * npartitions_per_node);
+ part_last = part_first + (npartitions_per_node - 1);
+
+ /* buffers (blocks) */
+
+ /* first/last buffer */
+ buff_first = BufferPartitionsRegistry->partitions[part_first].first_buffer;
+ buff_last = BufferPartitionsRegistry->partitions[part_last].last_buffer;
+
+ /* beginning of the first block, end of last block */
+ startptr = BufferBlocks + ((Size) buff_first * BLCKSZ);
+ endptr = BufferBlocks + ((Size) (buff_last + 1) * BLCKSZ);
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers for node %d not well aligned [%p,%p] aligned [%p,%p]",
+ n, startptr, endptr,
+ (char *) TYPEALIGN(numa_page_size, startptr),
+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ pg_numa_bind_to_node(startptr, endptr, n);
+
+ /* buffer descriptors */
+
+ /* beginning of the first descriptor, end of last descriptor */
+ startptr = (char *) &BufferDescriptors[buff_first];
+ endptr = (char *) &BufferDescriptors[buff_last] + 1;
+
+ /* print some warnings when the partitions are not aligned */
+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||
+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))
+ {
+ elog(WARNING, "buffers descriptors for node %d not well aligned [%p,%p] aligned [%p,%p]",
+ n, startptr, endptr,
+ (char *) TYPEALIGN(numa_page_size, startptr),
+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));
+ }
+
+ /* best effort: align the pointers, so that the mbind() works */
+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);
+
+ /* XXX or should we use pg_numa_move_to_node? */
+ pg_numa_bind_to_node(startptr, endptr, n);
+ }
+#endif
}
AssertCheckBufferPartitions();
}
+/*
+ * BufferPartitionsCalculate
+ * Pick number of buffer partitions for the number of nodes and
+ * MIN_BUFFER_PARTITIONS.
+ *
+ * Picks the smallest number of partitions higher thah MIN_BUFFER_PARTITIONS,
+ * such that all nodes have the same number of partitions.
+ *
+ * This is best-effort with respect to size of the partitions. It's possible
+ * the partitions are not a perfect multiple of page size, in which case
+ * we set location only for the part where that is possible. The buffers on
+ * the "boundary" may get located up on arbitrary nodes.
+ *
+ * The extra complexity of figuring out the right "partition size" is not
+ * worth it, and it can lead to some partitions being much smaller. This way
+ * we end up with partitions of almost exactly the same size (one BLCKSZ is
+ * the largest difference).
+ *
+ * We expect shared buffers to be much larger than page size (at least on
+ * system where NUMA is a relevant feature), so the number of "not located"
+ * buffers should be a negligible fraction. This only affects pages between
+ * partitions for different nodes, so (nodes-1) pages. This is certainly
+ * fine with 2MB huge pages, but even with 1GB pages it should be OK (as
+ * such systems should have humongous amounts of memory).
+ *
+ * It also means we don't need to worry about memory page size before knowing
+ * if huge pages got used (which we only learn during allocation).
+ */
+void
+BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ int nnodes,
+ nparts,
+ nparts_per_node;
+
+#if USE_LIBNUMA
+ nnodes = numa_num_configured_nodes();
+ nparts_per_node = 1; /* at least one partition per node */
+
+ while ((nparts_per_node * nnodes) < MIN_BUFFER_PARTITIONS)
+ nparts_per_node++;
+
+ nparts = (nnodes * nparts_per_node);
+#else
+ /* without NUMA, assume there's just one node */
+ nnodes = 1;
+ nparts = MIN_BUFFER_PARTITIONS;
+ nparts_per_node = MIN_BUFFER_PARTITIONS;
+#endif
+
+ if (num_nodes)
+ *num_nodes = nnodes;
+
+ if (num_partitions)
+ *num_partitions = nparts;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = nparts_per_node;
+}
+
/*
* BufferPartitionCount
* Returns the number of partitions created.
@@ -277,13 +464,14 @@ BufferPartitionCount(void)
* The returned information is first/last buffer, number of buffers.
*/
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
{
BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -293,3 +481,17 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+void
+BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node)
+{
+ if (num_nodes)
+ *num_nodes = BufferPartitionsRegistry->nnodes;
+
+ if (num_partitions)
+ *num_partitions = BufferPartitionsRegistry->npartitions;
+
+ if (num_partitions_per_node)
+ *num_partitions_per_node = BufferPartitionsRegistry->npartitions_per_node;
+}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index fdb5bad7910..53ef5239e8d 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,15 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f15e74198c5..2e71c04282c 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2724,6 +2724,12 @@
max => 'INT_MAX / 2',
},
+{ name => 'shared_buffers_numa', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
+ short_desc => 'Locate partitions of shared buffers (and descriptors) to NUMA nodes.',
+ variable => 'shared_buffers_numa',
+ boot_val => 'false',
+},
+
{ name => 'shared_memory_size', type => 'int', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows the size of the server\'s main shared memory area (rounded up to the nearest MB).',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_UNIT_MB | GUC_RUNTIME_COMPUTED',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..c0f79c779cc 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -142,6 +142,7 @@
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
# (change requires restart)
+#shared_buffers_numa = off # NUMA-aware partitioning
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB # min 64kB
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 1b668fe1d91..8fe4d4ab7e3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,13 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+extern PGDLLIMPORT int pg_numa_bind_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index e5a887b9969..e944cee2e91 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -365,10 +365,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -378,7 +378,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -416,8 +416,12 @@ extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
extern int BufferPartitionCount(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionsCalculate(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
+extern void BufferPartitionsParams(int *num_nodes, int *num_partitions,
+ int *num_partitions_per_node);
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 79a3f44747a..1cf09e8fb7c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -158,10 +158,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -170,7 +172,9 @@ typedef struct BufferPartition
/* an array of information about all partitions */
typedef struct BufferPartitions
{
+ int nnodes; /* number of NUMA nodes */
int npartitions; /* number of partitions */
+ int npartitions_per_node; /* for convenience */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -206,6 +210,7 @@ extern PGDLLIMPORT const PgAioHandleCallbacks aio_local_buffer_readv_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
+extern PGDLLIMPORT bool shared_buffers_numa;
/* in localbuf.c */
extern PGDLLIMPORT int NLocBuffer;
@@ -390,6 +395,9 @@ extern void MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied,
int32 *buffers_already_dirty,
int32 *buffers_skipped);
+/* in buf_init.c */
+extern int BufferGetNode(Buffer buffer);
+
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 8954669273a..11c8a4503da 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -118,6 +121,94 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ int ret;
+ struct bitmask *nodemask;
+
+ if (node < 0)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ nodemask = numa_allocate_nodemask();
+ if (nodemask == NULL)
+ {
+ errno = ENOMEM;
+ return -1;
+ }
+
+ numa_bitmask_setbit(nodemask, node);
+
+ /*
+ * MPOL_BIND places the pages strictly on the node, and MPOL_MF_MOVE migrates
+ * pages already faulted in to that node. If mbind() fails, leave the default
+ * placement in effect, and report the failure.
+ */
+ ret = mbind(startptr, (endptr - startptr),
+ MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
+
+ numa_free_nodemask(nodemask);
+
+ return ret;
+}
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
#else
/* Empty wrappers */
@@ -140,4 +231,25 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+int
+pg_numa_bind_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
+Size
+pg_numa_page_size(void)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
--
2.54.0
[text/x-patch] v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch (14.3K, ../[email protected]/6-v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch)
download | inline diff:
From 052ac2256afbba3f25f20f683449a0bbb0af4241 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 2 Jun 2026 22:09:33 +0200
Subject: [PATCH v20260605 2/6] Infrastructure for partitioning of shared
buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep or to make the
partitioning NUMA-aware).
The registry is a small array of BufferPartition entries in shared
memory, with partitions sized to be a fair share of shared buffers.
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
* The buffers are divided as evenly as possible, with the first couple
partitions possibly getting an extra buffer.
---
contrib/pg_buffercache/Makefile | 3 +-
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 142 ++++++++++++++++++
src/include/storage/buf_internals.h | 5 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 0e618f66aec..7fd5cdfc43d 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,8 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql \
+ pg_buffercache--1.7--1.8.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
new file mode 100644
index 00000000000..d62b8339bfc
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -0,0 +1,23 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index 11499550945..d2fa8ba53ba 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'
+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index bf2e6c97220..d678cb045ba 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -31,6 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -75,6 +76,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -871,3 +873,87 @@ pg_buffercache_mark_dirty_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of shared buffers.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 1407c930c56..e593b02e0ca 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -26,10 +26,12 @@ char *BufferBlocks;
ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsRegistry;
static void BufferManagerShmemRequest(void *arg);
static void BufferManagerShmemInit(void *arg);
static void BufferManagerShmemAttach(void *arg);
+static void BufferPartitionsInit(void);
const ShmemCallbacks BufferManagerShmemCallbacks = {
.request_fn = BufferManagerShmemRequest,
@@ -69,6 +71,9 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
+/* number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
/*
* Register shared memory area for the buffer pool.
@@ -97,6 +102,13 @@ BufferManagerShmemRequest(void *arg)
.ptr = (void **) &BufferIOCVArray,
);
+ ShmemRequestStruct(.name = "Buffer Partition Registry",
+ .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),
+ /* Align descriptors to a cacheline boundary. */
+ .alignment = PG_CACHE_LINE_SIZE,
+ .ptr = (void **) &BufferPartitionsRegistry,
+ );
+
/*
* The array used to sort to-be-checkpointed buffer ids is located in
* shared memory, to avoid having to allocate significant amounts of
@@ -119,6 +131,12 @@ BufferManagerShmemRequest(void *arg)
static void
BufferManagerShmemInit(void *arg)
{
+ /*
+ * Initialize the buffer partition registry first, before other parts
+ * have a chance to touch the memory.
+ */
+ BufferPartitionsInit();
+
/*
* Initialize all the buffer headers.
*/
@@ -151,3 +169,127 @@ BufferManagerShmemAttach(void *arg)
WritebackContextInit(&BackendWritebackContext,
&backend_flush_after);
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsRegistry->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsRegistry->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsRegistry->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsRegistry->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * BufferPartitionsInit
+ * Initialize registry of buffer partitions.
+ */
+static void
+BufferPartitionsInit(void)
+{
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;
+ int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[n];
+
+ int num_buffers = part_buffers;
+ if (n < remaining_buffers)
+ num_buffers += 1;
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+/*
+ * BufferPartitionCount
+ * Returns the number of partitions created.
+ */
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsRegistry->npartitions;
+}
+
+/*
+ * BufferPartitionGet
+ * Returns information about a partition at the provided index.
+ *
+ * The returned information is first/last buffer, number of buffers.
+ */
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 89615a254a3..e5a887b9969 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -411,9 +411,14 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsRegistry;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 6837b35fc6d..79a3f44747a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -155,6 +155,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8cf40c87043..ea756015249 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -365,6 +365,8 @@ BufferHeapTupleTableSlot
BufferLockMode
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.54.0
[text/x-patch] v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch (4.9K, ../[email protected]/7-v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch)
download | inline diff:
From 238935f06c793dcd0271423af83e039d2ee00120 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 3 Jun 2026 16:30:28 +0200
Subject: [PATCH v20260605 1/6] Add shmem_populate and shmem_interleave GUCs
- shmem_populate - Forces mmap() with MAP_POPULATE, which faults all
memory pages backing the shared memory segment.
- shmem_interleave - Applies NUMA interleaving on the whole shared
memory segment, to balance allocations between nodes.
---
src/backend/port/sysv_shmem.c | 47 +++++++++++++++++++++++
src/backend/utils/misc/guc_parameters.dat | 14 +++++++
src/include/miscadmin.h | 4 ++
3 files changed, 65 insertions(+)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 2e3886cf9fe..9eaff838a04 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -27,6 +27,10 @@
#include <sys/shm.h>
#include <sys/stat.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "portability/mem.h"
@@ -98,6 +102,10 @@ void *UsedShmemSegAddr = NULL;
static Size AnonymousShmemSize;
static void *AnonymousShmem = NULL;
+/* GUCs */
+bool shmem_populate = false; /* MAP_POPULATE */
+bool shmem_interleave = false; /* NUMA interleaving */
+
static void *InternalIpcMemoryCreate(IpcMemoryKey memKey, Size size);
static void IpcMemoryDetach(int status, Datum shmaddr);
static void IpcMemoryDelete(int status, Datum shmId);
@@ -604,6 +612,21 @@ CreateAnonymousSegment(Size *size)
int mmap_errno = 0;
int mmap_flags = MAP_SHARED | MAP_ANONYMOUS | MAP_HASSEMAPHORE;
+ /* If requested, populate the shared memory by MAP_POPULATE. */
+ if (shmem_populate)
+ mmap_flags |= MAP_POPULATE;
+
+#ifdef USE_LIBNUMA
+ /*
+ * If requested, interleave the shared memory by setting a memory policy
+ * before the mmap() call. This really matters only with MAP_POPULATE,
+ * because without page faults the memory does not actually get placed
+ * to the nodes. But without MAP_POPULATE it's virtually free.
+ */
+ if (shmem_interleave)
+ numa_set_interleave_mask(numa_all_nodes_ptr);
+#endif
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -665,6 +688,30 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+#ifdef USE_LIBNUMA
+ /*
+ * If set the policy to interleaving by numa_set_membind(), undo it now by
+ * setting the policy to localalloc. With MAP_POPULATE, all the pages were
+ * faulted and are now interleaved on the available nodes.
+ *
+ * To handle the case without MAP_POPULATE, apply the interleaving policy to
+ * the shared memory segment allocated by mmap() before touching it in any
+ * way, so that it gets placed on the correct node on first access.
+ *
+ * This matters especially with huge pages, where it's possible to run out
+ * of huge pages on some nodes and then crash. By explicitly interleaving
+ * the whole segment, that's much less likely.
+ */
+ if (shmem_interleave)
+ {
+ /* undo the policy set by numa_set_membind() earlier */
+ numa_set_localalloc();
+
+ /* set interleaving policy for not yet faulted memory */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+ }
+#endif
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index afaa058b046..f15e74198c5 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -740,6 +740,20 @@
ifdef => 'DEBUG_NODE_TESTS_ENABLED',
},
+{ name => 'debug_shmem_interleave', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Forces interleaving for the whole shared memory segment.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_interleave',
+ boot_val => 'false'
+},
+
+{ name => 'debug_shmem_populate', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'Populates (faults) the whole shared memory segment using MAP_POPULATE.',
+ flags => 'GUC_NOT_IN_SAMPLE',
+ variable => 'shmem_populate',
+ boot_val => 'false'
+},
+
{ name => 'debug_write_read_parse_plan_trees', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Set this to force all parse and plan trees to be passed through outfuncs.c/readfuncs.c, to facilitate catching errors and omissions in those modules.',
flags => 'GUC_NOT_IN_SAMPLE',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 7de0a115402..13bb29f8702 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -213,6 +213,10 @@ extern PGDLLIMPORT Oid MyDatabaseTableSpace;
extern PGDLLIMPORT bool MyDatabaseHasLoginEventTriggers;
+extern PGDLLIMPORT bool shmem_populate;
+extern PGDLLIMPORT bool shmem_interleave;
+
+
/*
* Date/Time Configuration
*
--
2.54.0
view thread (87+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox