From: Greg Burd <greg@burd.me>
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Andres Freund <andres@anarazel.de>
Cc: Alexey Makhmutov <a.makhmutov@postgrespro.ru>
Cc: Tomas Vondra <tomas@vondra.me>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>
Subject: Re: Adding basic NUMA awareness
Date: Tue, 15 Sep 2026 16:45:03 -0400
Message-ID: <dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com> (raw)
In-Reply-To: <b3c09a66-b2f6-49a6-a2fe-bbfe0cda3e6f@app.fastmail.com>
References: <5bc6d309-61fe-4d8b-9e1f-a2961564a559@vondra.me>
<05df16f8-025a-43cd-9636-3194464012ed@vondra.me>
<CAKZiRmxwN+qMpbijCLPix_y6mwSjgus2CPPj=1+uFo9fQG-Knw@mail.gmail.com>
<403c813e-7d63-4ee8-a95f-4e5c1e310f4d@vondra.me>
<uezi46xhhbvdjgdi6wl7iqgfcdh4jmnnyzbfovdcrck6ywqa7j@fj3yimxvekk6>
<0e1b997d-99c8-40f4-bc32-6c044bc7ed9a@vondra.me>
<zndz3dlmp7xlypczxjbwdfrey3masto6vuwpnzjvgunslprv25@rucfkdgfpb4p>
<rsjxzhnxxfq5i5yxv66mhinb42o3vzmqpkbfpexpkk5prreh2l@jyp73gsyyfzn>
<2db78610-b480-4aa0-a1b6-57f1c2dcb708@vondra.me>
<clx4zzd7kau4vvh5ynu5ssxg3jqfqzurgcbtotytzgzkhb3nis@qfl5xwv44yad>
<2e1be441-5608-4bc8-9ac4-6fad9a060db4@vondra.me>
<334f7644-6716-4e63-a7c4-13c5cbcdd210@vondra.me>
<CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay+mOEKs9rzCkegUA@mail.gmail.com>
<86e7e218-9013-4cdb-9ed2-dfda49640b4d@vondra.me>
<CAKZiRmzumFJ8+Bqz5mfX+QGkrKEaH32zz7kPgEShHWXE4QzyQw@mail.gmail.com>
<c3954e28-0c38-4df8-b76e-ec09f6c04021@vondra.me>
<CAKZiRmwsjwDatuV5JfUaAbkTxjnUaL_UK3SkFTJee-jo-u_N7Q@mail.gmail.com>
<2b6abd8f-3659-4be1-9499-b4f00f973f22@vondra.me>
<CAKZiRmzigM9XOz6m0K070ZVSmaKpjnVR4CZZENkfeH-rNv78wA@mail.gmail.com>
<CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com>
<CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com>
<CAKZiRmzT1M1fEFf5fRq2Qg3Vz0wLbdY+uNdF-0ORhDTUOUVhcA@mail.gmail.com>
<f78dfe7d-4a37-40ec-8e60-c2cc47c48364@app.fastmail.com>
<b3c09a66-b2f6-49a6-a2fe-bbfe0cda3e6f@app.fastmail.com>
I withdrew my batched clock sweep on its own thread. The piece worth
keeping belongs here: replacing the 0..5 usage_count with a 1-bit
HOT/COOL cooling state, on top of your partitioning.
I built your v20260605 0001-0006 plus Wartak's 0008 and added the
evictor, then measured it against the same tree with the stock evictor.
Where it mattered I ran each test with and without partitioning to
separate an evictor effect from a NUMA effect. r8i.metal-96xl, 6 nodes,
your harness.
One result holds up. Under memory pressure HOT/COOL keeps more of the
frequently-used index pages resident than the 0..5 clock, and it does so
by the same margin with or without partitioning:
stock 0..5 HOT/COOL delta
unpartitioned:
index buffers resident 107,941 111,623 +3.4%
index share of evictions 36.89% 36.07% -0.83pt
6-NUMA partitioned:
index buffers resident 108,029 111,556 +3.3%
index share of evictions 36.88% 36.08% -0.80pt
Workload: 8GB cold heap, three 850MB secondary indexes, s_b=2GB, zipfian
index probes at 192 clients, 5 runs alternated. The per-run numbers
don't overlap (stock 36.87-36.90, HOT/COOL 36.06-36.14). Under 0..5 the
cold-heap fetches inflate usage counts and heap pages outlast index
pages by a sweep or two; the 1-bit state demotes them on the next pass,
so reused index pages win a little more often.
Because the margin is identical partitioned and not, it's a property of
the evictor, not of NUMA. So it's a small win on its own on
single-socket boxes, and it doesn't overlap with what partitioning buys
you. Throughput on top of partitioning is a wash on seqconcurrscans and
pgbenchS, and I'm not claiming otherwise.
One thing unrelated to the evictor that's useful to you: the c=1
seqconcurrscans regression gets worse with more nodes, -12% at 2 and
-49% at 6. Each partition is s_b/nodes, so at 6 nodes a single-client
scan no longer fits where it did unpartitioned. It looks like it wants a
floor on partition size rather than one partition per node. Happy to
test a floor if you want to try one.
Two build nits, both one-liners: pg_buffercache--1.7--1.8.sql updates
the Makefile but not meson.build, so CREATE EXTENSION fails under meson;
and 0008 needs a rebase (applies to master only with fuzz, leaves a
stray token in the ClockSweep struct).
Only 0009 is the change to the hot/cool algorithm, the other 8 patches
are what was on this list before.
best.
-greg
Attachments:
[text/x-patch] v20260915a-0001-Add-shmem_populate-and-shmem_interleave-G.patch (4.8K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/2-v20260915a-0001-Add-shmem_populate-and-shmem_interleave-G.patch)
download | inline diff:
From 7d1878f1f8b84e9e1e2e06ec65dd174de53f2a26 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Wed, 3 Jun 2026 16:30:28 +0200
Subject: [PATCH v20260915a 1/9] Add shmem_populate and shmem_interleave GUCs
- shmem_populate - Forces mmap() with MAP_POPULATE, which faults all
memory pages backing the shared memory segment.
- shmem_interleave - Applies NUMA interleaving on the whole shared
memory segment, to balance allocations between nodes.
---
src/backend/port/sysv_shmem.c | 47 +++++++++++++++++++++++
src/backend/utils/misc/guc_parameters.dat | 14 +++++++
src/include/miscadmin.h | 4 ++
3 files changed, 65 insertions(+)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.cindex 2e3886cf..9eaff838 100644--- a/src/backend/port/sysv_shmem.c+++ b/src/backend/port/sysv_shmem.c@@ -27,6 +27,10 @@
#include <sys/shm.h>
#include <sys/stat.h>
+#ifdef USE_LIBNUMA+#include <numa.h>+#endif+
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "portability/mem.h"
@@ -98,6 +102,10 @@ void *UsedShmemSegAddr = NULL;
static Size AnonymousShmemSize;
static void *AnonymousShmem = NULL;
+/* GUCs */+bool shmem_populate = false; /* MAP_POPULATE */+bool shmem_interleave = false; /* NUMA interleaving */+
static void *InternalIpcMemoryCreate(IpcMemoryKey memKey, Size size);
static void IpcMemoryDetach(int status, Datum shmaddr);
static void IpcMemoryDelete(int status, Datum shmId);
@@ -604,6 +612,21 @@ CreateAnonymousSegment(Size *size)
int mmap_errno = 0;
int mmap_flags = MAP_SHARED | MAP_ANONYMOUS | MAP_HASSEMAPHORE;
+ /* If requested, populate the shared memory by MAP_POPULATE. */+ if (shmem_populate)+ mmap_flags |= MAP_POPULATE;++#ifdef USE_LIBNUMA+ /*+ * If requested, interleave the shared memory by setting a memory policy+ * before the mmap() call. This really matters only with MAP_POPULATE,+ * because without page faults the memory does not actually get placed+ * to the nodes. But without MAP_POPULATE it's virtually free.+ */+ if (shmem_interleave)+ numa_set_interleave_mask(numa_all_nodes_ptr);+#endif+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -665,6 +688,30 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+#ifdef USE_LIBNUMA+ /*+ * If set the policy to interleaving by numa_set_membind(), undo it now by+ * setting the policy to localalloc. With MAP_POPULATE, all the pages were+ * faulted and are now interleaved on the available nodes.+ *+ * To handle the case without MAP_POPULATE, apply the interleaving policy to+ * the shared memory segment allocated by mmap() before touching it in any+ * way, so that it gets placed on the correct node on first access.+ *+ * This matters especially with huge pages, where it's possible to run out+ * of huge pages on some nodes and then crash. By explicitly interleaving+ * the whole segment, that's much less likely.+ */+ if (shmem_interleave)+ {+ /* undo the policy set by numa_set_membind() earlier */+ numa_set_localalloc();++ /* set interleaving policy for not yet faulted memory */+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);+ }+#endif+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.datindex c57441f7..693bad94 100644--- a/src/backend/utils/misc/guc_parameters.dat+++ b/src/backend/utils/misc/guc_parameters.dat@@ -738,6 +738,20 @@
ifdef => 'DEBUG_NODE_TESTS_ENABLED',
},
+{ name => 'debug_shmem_interleave', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',+ short_desc => 'Forces interleaving for the whole shared memory segment.',+ flags => 'GUC_NOT_IN_SAMPLE',+ variable => 'shmem_interleave',+ boot_val => 'false'+},++{ name => 'debug_shmem_populate', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',+ short_desc => 'Populates (faults) the whole shared memory segment using MAP_POPULATE.',+ flags => 'GUC_NOT_IN_SAMPLE',+ variable => 'shmem_populate',+ boot_val => 'false'+},+
{ name => 'debug_write_read_parse_plan_trees', type => 'bool', context => 'PGC_SUSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Set this to force all parse and plan trees to be passed through outfuncs.c/readfuncs.c, to facilitate catching errors and omissions in those modules.',
flags => 'GUC_NOT_IN_SAMPLE',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.hindex 0fc59af0..9d01f40b 100644--- a/src/include/miscadmin.h+++ b/src/include/miscadmin.h@@ -213,6 +213,10 @@ extern PGDLLIMPORT Oid MyDatabaseTableSpace;
extern PGDLLIMPORT bool MyDatabaseHasLoginEventTriggers;
+extern PGDLLIMPORT bool shmem_populate;+extern PGDLLIMPORT bool shmem_interleave;++
/*
* Date/Time Configuration
*
--
2.50.1
[text/x-patch] v20260915a-0002-Infrastructure-for-partitioning-of-shared.patch (14.3K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/3-v20260915a-0002-Infrastructure-for-partitioning-of-shared.patch)
download | inline diff:
From b36f700141bdbc6417723f1fe54590f09119ebd1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Tue, 2 Jun 2026 22:09:33 +0200
Subject: [PATCH v20260915a 2/9] Infrastructure for partitioning of shared
buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep or to make the
partitioning NUMA-aware).
The registry is a small array of BufferPartition entries in shared
memory, with partitions sized to be a fair share of shared buffers.
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
* The buffers are divided as evenly as possible, with the first couple
partitions possibly getting an extra buffer.
---
contrib/pg_buffercache/Makefile | 3 +-
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 142 ++++++++++++++++++
src/include/storage/buf_internals.h | 5 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefileindex 0e618f66..7fd5cdfc 100644--- a/contrib/pg_buffercache/Makefile+++ b/contrib/pg_buffercache/Makefile@@ -9,7 +9,8 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql \+ pg_buffercache--1.7--1.8.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sqlnew file mode 100644index 00000000..d62b8339--- /dev/null+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql@@ -0,0 +1,23 @@+-- complain if script is sourced in psql, rather than via CREATE EXTENSION+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit++-- Register the new functions.+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()+RETURNS SETOF RECORD+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'+LANGUAGE C PARALLEL SAFE;++-- Create a view for convenient access.+CREATE VIEW pg_buffercache_partitions AS+ SELECT P.* FROM pg_buffercache_partitions() AS P+ (partition integer, -- partition index+ num_buffers integer, -- number of buffers in the partition+ first_buffer integer, -- first buffer of partition+ last_buffer integer); -- last buffer of partition++-- Don't want these to be available to public.+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;++GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.controlindex 11499550..d2fa8ba5 100644--- a/contrib/pg_buffercache/pg_buffercache.control+++ b/contrib/pg_buffercache/pg_buffercache.control@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.cindex 51045599..6c9838b4 100644--- a/contrib/pg_buffercache/pg_buffercache_pages.c+++ b/contrib/pg_buffercache/pg_buffercache_pages.c@@ -31,6 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -77,6 +78,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_mark_dirty_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -922,3 +924,87 @@ pg_buffercache_mark_dirty_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
++/*+ * Inquire about partitioning of shared buffers.+ */+Datum+pg_buffercache_partitions(PG_FUNCTION_ARGS)+{+ FuncCallContext *funcctx;+ MemoryContext oldcontext;+ TupleDesc tupledesc;+ TupleDesc expected_tupledesc;+ HeapTuple tuple;+ Datum result;++ if (SRF_IS_FIRSTCALL())+ {+ funcctx = SRF_FIRSTCALL_INIT();++ /* Switch context when allocating stuff to be used in later calls */+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);++ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)+ elog(ERROR, "return type must be a row type");++ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)+ elog(ERROR, "incorrect number of output arguments");++ /* Construct a tuple descriptor for the result rows. */+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",+ INT4OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",+ INT4OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",+ INT4OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",+ INT4OID, -1, 0);++ funcctx->user_fctx = BlessTupleDesc(tupledesc);++ /* Return to original context when allocating transient memory */+ MemoryContextSwitchTo(oldcontext);++ /* Set max calls and remember the user function context. */+ funcctx->max_calls = BufferPartitionCount();+ }++ funcctx = SRF_PERCALL_SETUP();++ if (funcctx->call_cntr < funcctx->max_calls)+ {+ uint32 i = funcctx->call_cntr;++ int num_buffers,+ first_buffer,+ last_buffer;++ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];++ BufferPartitionGet(i, &num_buffers,+ &first_buffer, &last_buffer);++ values[0] = Int32GetDatum(i);+ nulls[0] = false;++ values[1] = Int32GetDatum(num_buffers);+ nulls[1] = false;++ values[2] = Int32GetDatum(first_buffer);+ nulls[2] = false;++ values[3] = Int32GetDatum(last_buffer);+ nulls[3] = false;++ /* Build and return the tuple. */+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);+ result = HeapTupleGetDatum(tuple);++ SRF_RETURN_NEXT(funcctx, result);+ }+ else+ SRF_RETURN_DONE(funcctx);+}diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.cindex 1407c930..e593b02e 100644--- a/src/backend/storage/buffer/buf_init.c+++ b/src/backend/storage/buffer/buf_init.c@@ -26,10 +26,12 @@ char *BufferBlocks;
ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsRegistry;
static void BufferManagerShmemRequest(void *arg);
static void BufferManagerShmemInit(void *arg);
static void BufferManagerShmemAttach(void *arg);
+static void BufferPartitionsInit(void);
const ShmemCallbacks BufferManagerShmemCallbacks = {
.request_fn = BufferManagerShmemRequest,
@@ -69,6 +71,9 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
+/* number of buffer partitions */+#define NUM_CLOCK_SWEEP_PARTITIONS 4+
/*
* Register shared memory area for the buffer pool.
@@ -97,6 +102,13 @@ BufferManagerShmemRequest(void *arg)
.ptr = (void **) &BufferIOCVArray,
);
+ ShmemRequestStruct(.name = "Buffer Partition Registry",+ .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),+ /* Align descriptors to a cacheline boundary. */+ .alignment = PG_CACHE_LINE_SIZE,+ .ptr = (void **) &BufferPartitionsRegistry,+ );+
/*
* The array used to sort to-be-checkpointed buffer ids is located in
* shared memory, to avoid having to allocate significant amounts of
@@ -119,6 +131,12 @@ BufferManagerShmemRequest(void *arg)
static void
BufferManagerShmemInit(void *arg)
{
+ /*+ * Initialize the buffer partition registry first, before other parts+ * have a chance to touch the memory.+ */+ BufferPartitionsInit();+
/*
* Initialize all the buffer headers.
*/
@@ -151,3 +169,127 @@ BufferManagerShmemAttach(void *arg)
WritebackContextInit(&BackendWritebackContext,
&backend_flush_after);
}
++/*+ * Sanity checks of buffers partitions - there must be no gaps, it must cover+ * the whole range of buffers, etc.+ */+static void+AssertCheckBufferPartitions(void)+{+#ifdef USE_ASSERT_CHECKING+ int num_buffers = 0;++ Assert(BufferPartitionsRegistry->npartitions > 0);++ for (int i = 0; i < BufferPartitionsRegistry->npartitions; i++)+ {+ BufferPartition *part = &BufferPartitionsRegistry->partitions[i];++ /*+ * We can get a single-buffer partition, if the sizing forces the last+ * partition to be just one buffer. But it's unlikely (and+ * undesirable).+ */+ Assert(part->first_buffer <= part->last_buffer);+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);++ num_buffers += part->num_buffers;++ /*+ * The first partition needs to start on buffer 0. Later partitions+ * need to be contiguous, without skipping any buffers.+ */+ if (i == 0)+ {+ Assert(part->first_buffer == 0);+ }+ else+ {+ BufferPartition *prev = &BufferPartitionsRegistry->partitions[i - 1];++ Assert((part->first_buffer - 1) == prev->last_buffer);+ }++ /* the last partition needs to end on buffer (NBuffers - 1) */+ if (i == (BufferPartitionsRegistry->npartitions - 1))+ {+ Assert(part->last_buffer == (NBuffers - 1));+ }+ }++ Assert(num_buffers == NBuffers);+#endif+}++/*+ * BufferPartitionsInit+ * Initialize registry of buffer partitions.+ */+static void+BufferPartitionsInit(void)+{+ int buffer = 0;++ /* number of buffers per partition (make sure to not overflow) */+ int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;+ int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;++ BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;++ for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)+ {+ BufferPartition *part = &BufferPartitionsRegistry->partitions[n];++ int num_buffers = part_buffers;+ if (n < remaining_buffers)+ num_buffers += 1;++ remaining_buffers -= num_buffers;++ Assert((num_buffers > 0) && (num_buffers <= part_buffers));+ Assert((buffer >= 0) && (buffer < NBuffers));++ part->num_buffers = num_buffers;+ part->first_buffer = buffer;+ part->last_buffer = buffer + (num_buffers - 1);++ buffer += num_buffers;+ }++ AssertCheckBufferPartitions();+}++/*+ * BufferPartitionCount+ * Returns the number of partitions created.+ */+int+BufferPartitionCount(void)+{+ return BufferPartitionsRegistry->npartitions;+}++/*+ * BufferPartitionGet+ * Returns information about a partition at the provided index.+ *+ * The returned information is first/last buffer, number of buffers.+ */+void+BufferPartitionGet(int idx, int *num_buffers,+ int *first_buffer, int *last_buffer)+{+ if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))+ {+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];++ *num_buffers = part->num_buffers;+ *first_buffer = part->first_buffer;+ *last_buffer = part->last_buffer;++ return;+ }++ elog(ERROR, "invalid partition index");+}diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.hindex e4ff5619..d6846b1f 100644--- a/src/include/storage/buf_internals.h+++ b/src/include/storage/buf_internals.h@@ -411,9 +411,14 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsRegistry;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
+extern int BufferPartitionCount(void);+extern void BufferPartitionGet(int idx, int *num_buffers,+ int *first_buffer, int *last_buffer);+
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.hindex 6837b35f..79a3f447 100644--- a/src/include/storage/bufmgr.h+++ b/src/include/storage/bufmgr.h@@ -155,6 +155,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*+ * information about one partition of shared buffers+ *+ * first/last buffer - the values are inclusive+ */+typedef struct BufferPartition+{+ int num_buffers; /* number of buffers */+ int first_buffer; /* first buffer of partition */+ int last_buffer; /* last buffer of partition */+} BufferPartition;++/* an array of information about all partitions */+typedef struct BufferPartitions+{+ int npartitions; /* number of partitions */+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];+} BufferPartitions;+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 16d17d8f..66809af9 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -363,6 +363,8 @@ BufferHeapTupleTableSlot
BufferLockMode
BufferLookupEnt
BufferManagerRelation
+BufferPartition+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.50.1
[text/x-patch] v20260915a-0003-NUMA-shared-buffers-partitioning.patch (26.6K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/4-v20260915a-0003-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From e1194e13213056946a0ad2abba9af4c76029281a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Tue, 2 Jun 2026 23:27:13 +0200
Subject: [PATCH v20260915a 3/9] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc.
In cases like pre-warming a database from a single worker (e.g. using
pg_prewarm), we may end up with severely unbalanced memory distribution
(with most memory located on a single NUMA node). Unbalanced allocation
may put a lot of pressure on the memory system on a small number of NUMA
nodes, limiting the bandwidth etc.
With zone_reclaim, the kernel would eventually move some of the memory
to other nodes, but that tends to take a long time and is unpredictable.
This change forces even distribution of shared buffers on all NUMA
nodes, improving predictability, reducing the time needed for warmup
during benchmarking, etc. It's also less dependent on what the CPU
scheduler decides to do (which cores get used for the warmup.)
The effect is similar to
numactl --interleave=all
in that the buffers are distributed on the NUMA nodes evenly, but
there's also a number of important differences.
Firstly, it's applied only to shared buffers (and buffer descriptors),
not to the whole shared memory segment. It's possible to enable memory
interleaving using the shmem_interleave GUC, introduced in an earlier
patch in this series.
NUMA works at the granularity of a memory page, which is typically
either 4K or 2MB (hugepage), but other sizes are possible. For systems
where NUMA matters, we expect large amounts of memory (hundreds of
gigabytes) and hugepages enabled. But not necessarily.
The partitioning scheme is best-effort with respect to memory page size.
The shared buffers do not "align" with memory pages (i.e. a partition
may not end at the memory page boundary), in which case we simply locate
just the section of the partition with complete memory pages. This means
there may be ~one unmapped memory page between partitions. Considering
the expected amounts of memory, this is negligible, and the alternative
would be a significant amount of complexity to align the pages and
enforce "allowed" partition sizes.
Buffer descriptors are affected by this too, and the effect may be more
significant, simply because the descriptors are much smaller (~64B). So
the array is smaller, and a single 2MB memory page is worth ~32K buffer
descriptors. But with large systems it's still negligible.
The "buffer partitions" may not be 1:1 with NUMA nodes. We want to allow
clock-sweep partitioning even on non-NUMA systems, or when running only
on a small number of NUMA nodes. There's a minimal number of partitions
(default: 4), and a node may get multiple partitions. Nodes always get
the same number of partitions (e.g. with 3 NUMA nodes there will be 6
partitions in total, as each node gets 2 partitions).
The feature is enabled by dshared_buffers_numa GUC (default: false).
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/buf_init.c | 244 ++++++++++++++++--
src/backend/storage/buffer/freelist.c | 9 +
src/backend/utils/misc/guc_parameters.dat | 6 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/port/pg_numa.h | 7 +
src/include/storage/buf_internals.h | 16 +-
src/include/storage/bufmgr.h | 8 +
src/port/pg_numa.c | 112 ++++++++
10 files changed, 392 insertions(+), 36 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sqlindex d62b8339..a6e49fd1 100644--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer); -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.cindex 6c9838b4..46b6b85a 100644--- a/contrib/pg_buffercache/pg_buffercache_pages.c+++ b/contrib/pg_buffercache/pg_buffercache_pages.c@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -955,11 +955,13 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",+ INT4OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -977,28 +979,32 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,+ int numa_node,+ num_buffers,
first_buffer,
last_buffer;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
+ values[4] = Int32GetDatum(last_buffer);+ nulls[4] = false;+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.cindex e593b02e..1f93a31d 100644--- a/src/backend/storage/buffer/buf_init.c+++ b/src/backend/storage/buffer/buf_init.c@@ -14,12 +14,20 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA+#include <numa.h>+#include <numaif.h>+#endif++#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proclist.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
+#include "utils/guc_hooks.h"+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -71,9 +79,12 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
* multiple times. Check the PrivateRefCount infrastructure in bufmgr.c.
*/
-/* number of buffer partitions */-#define NUM_CLOCK_SWEEP_PARTITIONS 4+/*+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.+ */+#define MIN_BUFFER_PARTITIONS 4+bool shared_buffers_numa = false;
/*
* Register shared memory area for the buffer pool.
@@ -81,6 +92,10 @@ const ShmemCallbacks BufferManagerShmemCallbacks = {
static void
BufferManagerShmemRequest(void *arg)
{
+ int nparts;++ BufferPartitionsCalculate(NULL, &nparts, NULL);+
ShmemRequestStruct(.name = "Buffer Descriptors",
.size = NBuffers * sizeof(BufferDescPadded),
/* Align descriptors to a cacheline boundary. */
@@ -103,7 +118,7 @@ BufferManagerShmemRequest(void *arg)
);
ShmemRequestStruct(.name = "Buffer Partition Registry",
- .size = NUM_CLOCK_SWEEP_PARTITIONS * sizeof(BufferPartition),+ .size = nparts * sizeof(BufferPartition),
/* Align descriptors to a cacheline boundary. */
.alignment = PG_CACHE_LINE_SIZE,
.ptr = (void **) &BufferPartitionsRegistry,
@@ -134,6 +149,10 @@ BufferManagerShmemInit(void *arg)
/*
* Initialize the buffer partition registry first, before other parts
* have a chance to touch the memory.
+ *+ * Also moves memory to different NUMA nodes (if enabled by a GUC).+ * Do this before the loop that initializes buffer headers etc. which+ * may fault some of the memory pages etc.
*/
BufferPartitionsInit();
@@ -231,35 +250,203 @@ BufferPartitionsInit(void)
{
int buffer = 0;
- /* number of buffers per partition (make sure to not overflow) */- int part_buffers = NBuffers / NUM_CLOCK_SWEEP_PARTITIONS;- int remaining_buffers = NBuffers % NUM_CLOCK_SWEEP_PARTITIONS;+ int nnodes,+ npartitions,+ npartitions_per_node;- BufferPartitionsRegistry->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;+ int buffers_per_partition,+ buffers_remaining;- for (int n = 0; n < BufferPartitionsRegistry->npartitions; n++)- {- BufferPartition *part = &BufferPartitionsRegistry->partitions[n];+ /* calculate partitioning parameters */+ BufferPartitionsCalculate(&nnodes, &npartitions, &npartitions_per_node);++ /* paranoia */+ Assert(nnodes > 0);+ Assert(npartitions >= MIN_BUFFER_PARTITIONS);+ Assert((npartitions % nnodes) == 0);+ Assert((npartitions_per_node * nnodes) == npartitions);- int num_buffers = part_buffers;- if (n < remaining_buffers)- num_buffers += 1;+ BufferPartitionsRegistry->nnodes = nnodes;+ BufferPartitionsRegistry->npartitions = npartitions;+ BufferPartitionsRegistry->npartitions_per_node = npartitions_per_node;- remaining_buffers -= num_buffers;+ /* regular partition size, the first couple get an extra buffer */+ buffers_per_partition = (NBuffers / npartitions);+ buffers_remaining = (NBuffers % buffers_per_partition);- Assert((num_buffers > 0) && (num_buffers <= part_buffers));- Assert((buffer >= 0) && (buffer < NBuffers));+ /* should have all the buffers */+ Assert((buffers_per_partition * npartitions + buffers_remaining) == NBuffers);- part->num_buffers = num_buffers;- part->first_buffer = buffer;- part->last_buffer = buffer + (num_buffers - 1);+ /*+ * Now walk the partitions, and set the buffer range. Optionally, place+ * the partitions on a given node (for all partitions at once).+ */+ for (int n = 0; n < nnodes; n++)+ {+ for (int p = 0; p < npartitions_per_node; p++)+ {+ int idx = (n * npartitions_per_node) + p;+ BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];++ /*+ * Assign to the NUMA node, but only with shared_buffers_numa=on.+ *+ * XXX we should get an actual node ID from the mask, in case the+ * task is restricted to only some nodes.+ */+ part->numa_node = (shared_buffers_numa) ? n : -1;++ /* The first couple partitions may get an extra buffer. */+ part->num_buffers = buffers_per_partition;+ if (idx < buffers_remaining)+ part->num_buffers += 1;++ /* remember the buffer range */+ part->first_buffer = buffer;+ part->last_buffer = buffer + (part->num_buffers - 1);++ /* remember start of the next partition */+ buffer += part->num_buffers;+ }- buffer += num_buffers;+#ifdef USE_LIBNUMA+ /*+ * Now try to locate buffers and buffer descriptors to the node (all+ * partitions for the node at once).+ */+ if (shared_buffers_numa)+ {+ Size numa_page_size = pg_numa_page_size();++ int part_first,+ part_last,+ buff_first,+ buff_last;++ char *startptr,+ *endptr;++ /* first/last partition for this node */+ part_first = (n * npartitions_per_node);+ part_last = part_first + (npartitions_per_node - 1);++ /* buffers (blocks) */++ /* first/last buffer */+ buff_first = BufferPartitionsRegistry->partitions[part_first].first_buffer;+ buff_last = BufferPartitionsRegistry->partitions[part_last].last_buffer;++ /* beginning of the first block, end of last block */+ startptr = BufferBlocks + ((Size) buff_first * BLCKSZ);+ endptr = BufferBlocks + ((Size) (buff_last + 1) * BLCKSZ);++ /* print some warnings when the partitions are not aligned */+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))+ {+ elog(WARNING, "buffers for node %d not well aligned [%p,%p] aligned [%p,%p]",+ n, startptr, endptr,+ (char *) TYPEALIGN(numa_page_size, startptr),+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));+ }++ /* best effort: align the pointers, so that the mbind() works */+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);++ /* XXX or should we use pg_numa_move_to_node? */+ pg_numa_bind_to_node(startptr, endptr, n);++ /* buffer descriptors */++ /* beginning of the first descriptor, end of last descriptor */+ startptr = (char *) &BufferDescriptors[buff_first];+ endptr = (char *) &BufferDescriptors[buff_last] + 1;++ /* print some warnings when the partitions are not aligned */+ if ((startptr != (char *) TYPEALIGN(numa_page_size, startptr)) ||+ (endptr != (char *) TYPEALIGN_DOWN(numa_page_size, endptr)))+ {+ elog(WARNING, "buffers descriptors for node %d not well aligned [%p,%p] aligned [%p,%p]",+ n, startptr, endptr,+ (char *) TYPEALIGN(numa_page_size, startptr),+ (char *) TYPEALIGN_DOWN(numa_page_size, endptr));+ }++ /* best effort: align the pointers, so that the mbind() works */+ startptr = (char *) TYPEALIGN(numa_page_size, startptr);+ endptr = (char *) TYPEALIGN_DOWN(numa_page_size, endptr);++ /* XXX or should we use pg_numa_move_to_node? */+ pg_numa_bind_to_node(startptr, endptr, n);+ }+#endif
}
AssertCheckBufferPartitions();
}
+/*+ * BufferPartitionsCalculate+ * Pick number of buffer partitions for the number of nodes and+ * MIN_BUFFER_PARTITIONS.+ *+ * Picks the smallest number of partitions higher thah MIN_BUFFER_PARTITIONS,+ * such that all nodes have the same number of partitions.+ *+ * This is best-effort with respect to size of the partitions. It's possible+ * the partitions are not a perfect multiple of page size, in which case+ * we set location only for the part where that is possible. The buffers on+ * the "boundary" may get located up on arbitrary nodes.+ *+ * The extra complexity of figuring out the right "partition size" is not+ * worth it, and it can lead to some partitions being much smaller. This way+ * we end up with partitions of almost exactly the same size (one BLCKSZ is+ * the largest difference).+ *+ * We expect shared buffers to be much larger than page size (at least on+ * system where NUMA is a relevant feature), so the number of "not located"+ * buffers should be a negligible fraction. This only affects pages between+ * partitions for different nodes, so (nodes-1) pages. This is certainly+ * fine with 2MB huge pages, but even with 1GB pages it should be OK (as+ * such systems should have humongous amounts of memory).+ *+ * It also means we don't need to worry about memory page size before knowing+ * if huge pages got used (which we only learn during allocation).+ */+void+BufferPartitionsCalculate(int *num_nodes, int *num_partitions,+ int *num_partitions_per_node)+{+ int nnodes,+ nparts,+ nparts_per_node;++#if USE_LIBNUMA+ nnodes = numa_num_configured_nodes();+ nparts_per_node = 1; /* at least one partition per node */++ while ((nparts_per_node * nnodes) < MIN_BUFFER_PARTITIONS)+ nparts_per_node++;++ nparts = (nnodes * nparts_per_node);+#else+ /* without NUMA, assume there's just one node */+ nnodes = 1;+ nparts = MIN_BUFFER_PARTITIONS;+ nparts_per_node = MIN_BUFFER_PARTITIONS;+#endif++ if (num_nodes)+ *num_nodes = nnodes;++ if (num_partitions)+ *num_partitions = nparts;++ if (num_partitions_per_node)+ *num_partitions_per_node = nparts_per_node;+}+
/*
* BufferPartitionCount
* Returns the number of partitions created.
@@ -277,13 +464,14 @@ BufferPartitionCount(void)
* The returned information is first/last buffer, number of buffers.
*/
void
-BufferPartitionGet(int idx, int *num_buffers,+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsRegistry->npartitions))
{
BufferPartition *part = &BufferPartitionsRegistry->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -293,3 +481,17 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
++void+BufferPartitionsParams(int *num_nodes, int *num_partitions,+ int *num_partitions_per_node)+{+ if (num_nodes)+ *num_nodes = BufferPartitionsRegistry->nnodes;++ if (num_partitions)+ *num_partitions = BufferPartitionsRegistry->npartitions;++ if (num_partitions_per_node)+ *num_partitions_per_node = BufferPartitionsRegistry->npartitions_per_node;+}diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex fdb5bad7..53ef5239 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -15,6 +15,15 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA+#include <sched.h>+#endif++#ifdef USE_LIBNUMA+#include <numa.h>+#include <numaif.h>+#endif+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.datindex 693bad94..95910bcb 100644--- a/src/backend/utils/misc/guc_parameters.dat+++ b/src/backend/utils/misc/guc_parameters.dat@@ -2746,6 +2746,12 @@
max => 'INT_MAX / 2',
},
+{ name => 'shared_buffers_numa', type => 'bool', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',+ short_desc => 'Locate partitions of shared buffers (and descriptors) to NUMA nodes.',+ variable => 'shared_buffers_numa',+ boot_val => 'false',+},+
{ name => 'shared_memory_size', type => 'int', context => 'PGC_INTERNAL', group => 'PRESET_OPTIONS',
short_desc => 'Shows the size of the server\'s main shared memory area (rounded up to the nearest MB).',
flags => 'GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_UNIT_MB | GUC_RUNTIME_COMPUTED',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sampleindex e759f06b..c3406b5e 100644--- a/src/backend/utils/misc/postgresql.conf.sample+++ b/src/backend/utils/misc/postgresql.conf.sample@@ -142,6 +142,7 @@
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
# (change requires restart)
+#shared_buffers_numa = off # NUMA-aware partitioning
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB # min 64kB
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.hindex 1b668fe1..8fe4d4ab 100644--- a/src/include/port/pg_numa.h+++ b/src/include/port/pg_numa.h@@ -17,6 +17,13 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);+extern PGDLLIMPORT int pg_numa_bind_to_node(char *startptr, char *endptr, int node);++extern PGDLLIMPORT int numa_flags;++#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.hindex d6846b1f..b082e36a 100644--- a/src/include/storage/buf_internals.h+++ b/src/include/storage/buf_internals.h@@ -365,10 +365,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit- * more noticeable on 32bit systems, we don't force the stride to be cache- * line sized on those. If somebody does actual performance testing, we can- * reevaluate.+ * probably all are 64bit these days. We force the stride to be cache line+ * sized even on 32bit systems, where the space wastage is be a bit more+ * noticeable, to allow partitioning of shared buffers (which requires the+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -378,7 +378,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -416,8 +416,12 @@ extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
extern int BufferPartitionCount(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionsCalculate(int *num_nodes, int *num_partitions,+ int *num_partitions_per_node);+extern void BufferPartitionsParams(int *num_nodes, int *num_partitions,+ int *num_partitions_per_node);
/* in localbuf.c */
extern PGDLLIMPORT BufferDesc *LocalBufferDescriptors;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.hindex 79a3f447..1cf09e8f 100644--- a/src/include/storage/bufmgr.h+++ b/src/include/storage/bufmgr.h@@ -158,10 +158,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -170,7 +172,9 @@ typedef struct BufferPartition
/* an array of information about all partitions */
typedef struct BufferPartitions
{
+ int nnodes; /* number of NUMA nodes */
int npartitions; /* number of partitions */
+ int npartitions_per_node; /* for convenience */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -206,6 +210,7 @@ extern PGDLLIMPORT const PgAioHandleCallbacks aio_local_buffer_readv_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
+extern PGDLLIMPORT bool shared_buffers_numa;
/* in localbuf.c */
extern PGDLLIMPORT int NLocBuffer;
@@ -390,6 +395,9 @@ extern void MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied,
int32 *buffers_already_dirty,
int32 *buffers_skipped);
+/* in buf_init.c */+extern int BufferGetNode(Buffer buffer);+
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.cindex 89546692..11c8a450 100644--- a/src/port/pg_numa.c+++ b/src/port/pg_numa.c@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"++int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -118,6 +121,94 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*+ * pg_numa_move_to_node+ * move memory to different NUMA nodes in larger chunks+ *+ * startptr - start of the region (should be aligned to page size)+ * endptr - end of the region (doesn't need to be aligned)+ * node - node to move the memory to+ *+ * The "startptr" is expected to be a multiple of system memory page size, as+ * determined by pg_numa_page_size.+ *+ * XXX We only expect to do this during startup, when the shared memory is+ * still being setup.+ */+void+pg_numa_move_to_node(char *startptr, char *endptr, int node)+{+ Size sz = (endptr - startptr);++ Assert((int64) startptr % pg_numa_page_size() == 0);++ /*+ * numa_tonode_memory does not actually cause a page fault, and thus does+ * not locate the memory on the node. So it's fast, at least compared to+ * pg_numa_query_pages, and does not make startup longer. But it also+ * means the expensive part happen later, on the first access.+ */+ numa_tonode_memory(startptr, sz, node);+}++int+pg_numa_bind_to_node(char *startptr, char *endptr, int node)+{+ int ret;+ struct bitmask *nodemask;++ if (node < 0)+ {+ errno = EINVAL;+ return -1;+ }++ nodemask = numa_allocate_nodemask();+ if (nodemask == NULL)+ {+ errno = ENOMEM;+ return -1;+ }++ numa_bitmask_setbit(nodemask, node);++ /*+ * MPOL_BIND places the pages strictly on the node, and MPOL_MF_MOVE migrates+ * pages already faulted in to that node. If mbind() fails, leave the default+ * placement in effect, and report the failure.+ */+ ret = mbind(startptr, (endptr - startptr),+ MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);++ numa_free_nodemask(nodemask);++ return ret;+}++Size+pg_numa_page_size(void)+{+ Size os_page_size;+ Size huge_page_size;++#ifdef WIN32+ SYSTEM_INFO sysinfo;++ GetSystemInfo(&sysinfo);+ os_page_size = sysinfo.dwPageSize;+#else+ os_page_size = sysconf(_SC_PAGESIZE);+#endif++ /* assume huge pages get used, unless HUGE_PAGES_OFF */+ if (huge_pages_status != HUGE_PAGES_OFF)+ GetHugePageSize(&huge_page_size, NULL);+ else+ huge_page_size = 0;++ return Max(os_page_size, huge_page_size);+}+
#else
/* Empty wrappers */
@@ -140,4 +231,25 @@ pg_numa_get_max_node(void)
return 0;
}
+void+pg_numa_move_to_node(char *startptr, char *endptr, int node)+{+ /* we don't expect to ever get here in builds without libnuma */+ Assert(false);+}++int+pg_numa_bind_to_node(char *startptr, char *endptr, int node)+{+ /* we don't expect to ever get here in builds without libnuma */+ Assert(false);+}++Size+pg_numa_page_size(void)+{+ /* we don't expect to ever get here in builds without libnuma */+ Assert(false);+}+
#endif
--
2.50.1
[text/x-patch] v20260915a-0004-clock-sweep-basic-partitioning.patch (33.9K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/5-v20260915a-0004-clock-sweep-basic-partitioning.patch)
download | inline diff:
From ab25d0c831435f96d8cbd110fe5b5bc1e1819915 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Tue, 2 Jun 2026 22:16:55 +0200
Subject: [PATCH v20260915a 4/9] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/bufmgr.c | 202 +++++++----
src/backend/storage/buffer/freelist.c | 333 ++++++++++++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 486 insertions(+), 104 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sqlindex a6e49fd1..92176fed 100644--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql@@ -14,7 +14,13 @@ CREATE VIEW pg_buffercache_partitions AS
numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition+ last_buffer integer, -- last buffer of partition++ -- clocksweep counters+ num_passes bigint, -- clocksweep passes+ next_buffer integer, -- next victim buffer for clocksweep+ total_allocs bigint, -- handled allocs (running total)+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.cindex 46b6b85a..b07fafda 100644--- a/contrib/pg_buffercache/pg_buffercache_pages.c+++ b/contrib/pg_buffercache/pg_buffercache_pages.c@@ -31,7 +31,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -963,6 +963,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",+ INT8OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",+ INT4OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",+ INT8OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -984,12 +992,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;++ uint32 complete_passes,+ next_victim_buffer,+ buffer_allocs;+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,+ &complete_passes, &next_victim_buffer,+ &buffer_total_allocs, &buffer_allocs);+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -1005,6 +1023,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(complete_passes);+ nulls[5] = false;++ values[6] = Int32GetDatum(next_victim_buffer);+ nulls[6] = false;++ values[7] = Int64GetDatum(buffer_total_allocs);+ nulls[7] = false;++ values[8] = Int64GetDatum(buffer_allocs);+ nulls[8] = false;+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.cindex 5c82865a..b58f59d5 100644--- a/src/backend/storage/buffer/bufmgr.c+++ b/src/backend/storage/buffer/bufmgr.c@@ -3840,33 +3840,34 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.- *- * This is called periodically by the background writer process.+ * Information saved between calls so we can determine the strategy point's+ * advance rate and avoid scanning already-cleaned buffers.
*
- * Returns true if it's appropriate for the bgwriter process to go into- * low-power hibernation mode. (This happens if the strategy clock-sweep- * has been "lapped" and no buffer allocations have occurred recently,- * or if the bgwriter has been effectively disabled by setting- * bgwriter_lru_maxpages to 0.)+ * XXX Does it actually make sense to split all of this information per+ * partition? For example, does per-partition advance rate mean anything?+ * Maybe we should have a global advance rate? Although, if we want to+ * keep enough clean buffers in each partition, maybe having per-partition+ * rates makes sense.
*/
-bool-BgBufferSync(WritebackContext *wb_context)+typedef struct BufferSyncPartition+{+ int prev_strategy_buf_id;+ uint32 prev_strategy_passes;+ int next_to_clean;+ uint32 next_passes;+} BufferSyncPartition;++static BufferSyncPartition *saved_info = NULL;+static bool saved_info_valid = false;++static bool+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,+ int partition, int recent_alloc_partition,+ BufferSyncPartition *saved)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;-- /*- * Information saved between calls so we can determine the strategy- * point's advance rate and avoid scanning already-cleaned buffers.- */- static bool saved_info_valid = false;- static int prev_strategy_buf_id;- static uint32 prev_strategy_passes;- static int next_to_clean;- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3894,25 +3895,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */+ int first_buffer;+ int num_buffers;+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);-- /* Report buffer alloc counts to pgstat */- PendingBgWriterStats.buf_alloc += recent_alloc;-- /*- * If we're not running the LRU scan, just stop after doing the stats- * stuff. We mark the saved state invalid so that we can recover sanely- * if LRU scan is turned back on later.- */- if (bgwriter_lru_maxpages <= 0)- {- saved_info_valid = false;- return true;- }+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3924,17 +3916,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;+ int32 passes_delta = strategy_passes - saved->prev_strategy_passes;- strategy_delta = strategy_buf_id - prev_strategy_buf_id;- strategy_delta += (long) passes_delta * NBuffers;+ strategy_delta = strategy_buf_id - saved->prev_strategy_buf_id;+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)+ if ((int32) (saved->next_passes - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;+ bufs_to_lap = strategy_buf_id - saved->next_to_clean;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3942,11 +3934,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&- next_to_clean >= strategy_buf_id)+ else if (saved->next_passes == strategy_passes &&+ saved->next_to_clean >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);+ bufs_to_lap = num_buffers - (saved->next_to_clean - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3966,9 +3958,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;- next_passes = strategy_passes;- bufs_to_lap = NBuffers;+ saved->next_to_clean = strategy_buf_id;+ saved->next_passes = strategy_passes;+ bufs_to_lap = num_buffers;
}
}
else
@@ -3982,15 +3974,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;- next_passes = strategy_passes;- bufs_to_lap = NBuffers;+ saved->next_to_clean = strategy_buf_id;+ saved->next_passes = strategy_passes;+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;- prev_strategy_passes = strategy_passes;- saved_info_valid = true;+ saved->prev_strategy_buf_id = strategy_buf_id;+ saved->prev_strategy_passes = strategy_passes;+ /* XXX this needs to happen only after all partitions */+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3998,9 +3991,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -4010,7 +4003,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -4018,10 +4011,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)- smoothed_alloc = recent_alloc;+ if (smoothed_alloc <= (float) recent_alloc_partition)+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -4048,7 +4041,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -4073,20 +4066,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,+ int sync_state = SyncOneBuffer(saved->next_to_clean, true,
wb_context);
- if (++next_to_clean >= NBuffers)+ if (++saved->next_to_clean >= (first_buffer + num_buffers))
{
- next_to_clean = 0;- next_passes++;+ saved->next_to_clean = first_buffer;+ saved->next_passes++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -4100,7 +4093,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -4130,8 +4123,83 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);+}++/*+ * BgBufferSync -- Write out some dirty buffers in the pool.+ *+ * This is called periodically by the background writer process.+ *+ * Returns true if it's appropriate for the bgwriter process to go into+ * low-power hibernation mode. (This happens if the strategy clock-sweep+ * has been "lapped" and no buffer allocations have occurred recently,+ * or if the bgwriter has been effectively disabled by setting+ * bgwriter_lru_maxpages to 0.)+ */+bool+BgBufferSync(WritebackContext *wb_context)+{+ /* info obtained from freelist.c */+ uint32 recent_alloc;+ uint32 recent_alloc_partition;+ int num_partitions;++ /* assume we can hibernate, any partition can set to false */+ bool hibernate = true;++ /* get the number of clocksweep partitions, and total alloc count */+ StrategySyncPrepare(&num_partitions, &recent_alloc);++ /* allocate space for per-partition information between calls */+ if (saved_info == NULL)+ {+ /*+ * XXX Not great it's using malloc(), but how else to allocate a+ * variable-length array?+ */+ saved_info = malloc(sizeof(BufferSyncPartition) * num_partitions);+ }++ /* Report buffer alloc counts to pgstat */+ PendingBgWriterStats.buf_alloc += recent_alloc;++ /* average alloc buffers per partition */+ recent_alloc_partition = (recent_alloc / num_partitions);++ /*+ * If we're not running the LRU scan, just stop after doing the stats+ * stuff. We mark the saved state invalid so that we can recover sanely+ * if LRU scan is turned back on later.+ */+ if (bgwriter_lru_maxpages <= 0)+ {+ saved_info_valid = false;+ return true;+ }++ /*+ * now process the clocksweep partitions, one by one, using the same+ * cleanup that we used for all buffers+ *+ * XXX Maybe we should randomize the order of partitions a bit, so that we+ * don't start from partition 0 all the time? Perhaps not entirely, but at+ * least pick a random starting point?+ */+ for (int partition = 0; partition < num_partitions; partition++)+ {+ /* hibernate if all partitions can hibernate */+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,+ partition, recent_alloc_partition,+ &saved_info[partition]);+ }++ /* now that we've scanned all partitions, mark the cached info as valid */+ saved_info_valid = true;+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex 53ef5239..2d565796 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -36,17 +36,28 @@
/*
- * The shared freelist control information.+ * Information about one partition of the ClockSweep (on a subset of buffers).+ *+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;+ slock_t clock_sweep_lock;++ /* range for this clock sweep partition */+ int32 node;+ int32 firstBuffer;+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *+ * XXX This is relative to firstBuffer, so needs to be offset properly.+ *+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -57,11 +68,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */+ pg_atomic_uint64 numTotalAllocs;++} ClockSweep;++/*+ * The shared freelist control information.+ */+typedef struct+{+ /* Spinlock: protects the values below */+ slock_t buffer_strategy_lock;+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
++ /* cached info about freelist partitioning */+ int num_nodes;+ int num_partitions;+ int num_partitions_per_node;++ /* clocksweep partitions */+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -108,6 +140,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -119,6 +152,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -126,14 +160,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);- if (victim >= NBuffers)+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -159,19 +193,118 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);+ SpinLockAcquire(&sweep->clock_sweep_lock);- wrapped = expected % NBuffers;+ wrapped = expected % sweep->numBuffers;- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;- SpinLockRelease(&StrategyControl->buffer_strategy_lock);+ sweep->completePasses++;+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;++ /*+ * Make sure we've calculated a buffer in the range of the partition. Buffer+ * IDs are 1-based, we're calculating 0-based indexes.+ */+ Assert((victim >= 0) && (victim < sweep->numBuffers));+ Assert(BufferIsValid(1 + sweep->firstBuffer + victim));++ return sweep->firstBuffer + victim;+}++/*+ * ClockSweepPartitionIndex+ * pick the clock-sweep partition to use based on PID and NUMA node+ *+ * With libnuma, use the NUMA node and PID to pick the partition. Otherwise+ * use just PID (as if there's a single NUMA node).+ *+ * XXX This should also check if buffers are NUMA-partitioned, not just if+ * compiled with libnuma.+ */+static int+ClockSweepPartitionIndex(void)+{+ int node = 0,+ index;+ pid_t pid = MyProcPid;;++ Assert(StrategyControl->num_partitions ==+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));++ /*+ * If buffers are NUMA-partitioned, determine the partition using the NUMA+ * node and PID. Without NUMA assume everything is a single NUMA node 0, and+ * we pick the partition based on PID.+ */+#ifdef USE_LIBNUMA+ if (shared_buffers_numa)+ {+ int cpu;++ /* XXX do we need to check sched_getcpu is available, somehow? */+ if ((cpu = sched_getcpu()) < 0)+ elog(ERROR, "sched_getcpu failed: %m");++ node = numa_node_of_cpu(cpu);+ }+#endif++ /*+ * We should't get unexpected NUMA nodes, not considered when setting up the+ * buffer partitions. It could happen if the allowed NUMA nodes get adjusted+ * at runtime, but at this point we just create partitions for all existing+ * nodes. We could plan for allowed partitions, but then what if those get+ * disabled, and the user allows some other partitions?+ */+ if ((node < 0) || (node > StrategyControl->num_nodes))+ elog(ERROR, "node out of range: %d > %u", node, StrategyControl->num_nodes);++ /*+ * Calculate the partition index. Nodes have the same number of partitions,+ * and we use the PID to pick one of those (for a given node). If there's+ * only a single partition per node, we can ignore PID and use node directly.+ */+ if (StrategyControl->num_partitions_per_node == 1)+ {+ /* fast-path */+ index = node;+ }+ else+ {+ /* use PID to pick one of node's partitions */+ index = (node * StrategyControl->num_partitions_per_node)+ + (pid % StrategyControl->num_partitions_per_node);+ }++ /* should have a valid partition index */+ Assert((index >= 0) && (index < StrategyControl->num_partitions));++ return index;+}++/*+ * ChooseClockSweep+ * pick a clocksweep partition based on NUMA node and PID+ *+ * Pick a partition mapped to the NUMA node the backend is currently running+ * on, and use PID if there are multiple partitions per node. Without NUMA+ * supported/enabled, use just PID.+ *+ * XXX Maybe we should do both the total and "per group" counts a power of+ * two? That'd allow using shifts instead of divisions in the calculation,+ * and that's cheaper. But how would that deal with odd number of nodes?+ */+static ClockSweep *+ChooseClockSweep(void)+{+ int index = ClockSweepPartitionIndex();++ return &StrategyControl->sweeps[index];
}
/*
@@ -242,10 +375,37 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* We count buffer allocation requests so that the bgwriter can estimate
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
+ *+ * XXX It's not quite right we call ChooseClockSweep twice - now, and then+ * a couple lines later (through ClockSweepTick). If the process moves+ * between CPUs / NUMA nodes in between, these call may pick different+ * partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);- /* Use the "clock sweep" algorithm to find a free buffer */+ /*+ * Use the "clock sweep" algorithm to find a free buffer+ *+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at+ * buffers from a single partition, aligned with the NUMA node. That means+ * a process "sweeps" only a fraction of buffers, even if the other buffers+ * are better candidates for eviction. Maybe there should be some logic to+ * "steal" buffers from other partitions or other nodes?+ *+ * XXX This only searches a single partition, which can result in "no+ * unpinned buffers available" even if there are buffers in other+ * partitions. Needs to scan partitions if needed, as a fallback.+ *+ * XXX Would that also mean we should have multiple bgwriters, one for each+ * node, or would one bgwriter still handle all nodes?+ *+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer+ * count for that one partition. In fact, this should not call ClockSweepTick+ * for every iteration. The call is likely quite expensive (does a lot+ * of stuff), and also may return a different partition on each call.+ * We should just do it once, and then do the for(;;) loop. And then+ * maybe advance to the next partition, until we scan through all of them.+ */
trycounter = NBuffers;
for (;;)
{
@@ -325,6 +485,48 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*+ * StrategySyncPrepare -- prepare for sync of all partitions+ *+ * Determine the number of clocksweep partitions, and calculate the recent+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync+ * to calculate average number of allocations per partition for the next+ * sync cycle.+ *+ * In addition it returns the count of recent buffer allocs, which is a total+ * summed from all partitions. The alloc counts are reset after being read,+ * as the partitions are walked.+ */+void+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)+{+ *num_buf_alloc = 0;+ *num_parts = StrategyControl->num_partitions;++ /*+ * We lock the partitions one by one, so not exacly in sync, but that+ * should be fine. We're only looking for heuristics anyway.+ */+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ ClockSweep *sweep = &StrategyControl->sweeps[i];++ /* XXX Do we need the lock, if we're only accessing atomics? Surely not. */+ /* XXX Are we ever calling this without num_buf_alloc? */+ SpinLockAcquire(&sweep->clock_sweep_lock);+ if (num_buf_alloc)+ {+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);++ /* include the count in the running total */+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);++ *num_buf_alloc += allocs;+ }+ SpinLockRelease(&sweep->clock_sweep_lock);+ }+}+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -332,37 +534,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer- * allocs if non-NULL pointers are passed. The alloc count is reset after- * being read.+ * the higher-order bits of nextVictimBuffer).+ *+ * This only considers a single clocksweep partition, as BgBufferSync looks+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)+StrategySyncStart(int partition, uint32 *complete_passes,+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);- result = nextVictimBuffer % NBuffers;+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));++ SpinLockAcquire(&sweep->clock_sweep_lock);+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);+ result = nextVictimBuffer % sweep->numBuffers;++ *first_buffer = sweep->firstBuffer;+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);- if (num_buf_alloc)- {- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);- }- SpinLockRelease(&StrategyControl->buffer_strategy_lock);- return result;+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));++ return sweep->firstBuffer + result;
}
/*
@@ -394,8 +603,14 @@ StrategyNotifyBgWriter(int bgwprocno)
static void
StrategyCtlShmemRequest(void *arg)
{
+ int num_partitions;++ /* get the number of buffer partitions */+ BufferPartitionsCalculate(NULL, &num_partitions, NULL);+
ShmemRequestStruct(.name = "Buffer Strategy Status",
- .size = sizeof(BufferStrategyControl),+ .size = offsetof(BufferStrategyControl, sweeps) ++ mul_size(num_partitions, sizeof(ClockSweep)),
.ptr = (void **) &StrategyControl
);
}
@@ -408,12 +623,42 @@ StrategyCtlShmemInit(void *arg)
{
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);+ /* Remember the number of partitions */+ BufferPartitionsParams(&StrategyControl->num_nodes,+ &StrategyControl->num_partitions,+ &StrategyControl->num_partitions_per_node);++ /* Initialize the clock sweep pointers (for all partitions) */+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ int node,+ num_buffers,+ first_buffer,+ last_buffer;++ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);- /* Clear statistics */- StrategyControl->completePasses = 0;- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);++ /* get info about the buffer partition */+ BufferPartitionGet(i, &node, &num_buffers,+ &first_buffer, &last_buffer);++ /*+ * FIXME This may not quite right, because if NBuffers is not a+ * perfect multiple of numBuffers, the last partition will have+ * numBuffers set too high. buf_init handles this by tracking the+ * remaining number of buffers, and not overflowing.+ */+ StrategyControl->sweeps[i].node = node;+ StrategyControl->sweeps[i].numBuffers = num_buffers;+ StrategyControl->sweeps[i].firstBuffer = first_buffer;++ /* Clear statistics */+ StrategyControl->sweeps[i].completePasses = 0;+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -777,3 +1022,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
++void+ClockSweepPartitionGetInfo(int idx,+ uint32 *complete_passes, uint32 *next_victim_buffer,+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)+{+ ClockSweep *sweep = &StrategyControl->sweeps[idx];++ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));++ /* get the clocksweep stats */+ *complete_passes = sweep->completePasses;+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);++ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);++ /* calculate the actual buffer ID */+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);+}diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.hindex b082e36a..f9736c27 100644--- a/src/include/storage/buf_internals.h+++ b/src/include/storage/buf_internals.h@@ -600,7 +600,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);+extern int StrategySyncStart(int partition, uint32 *complete_passes,+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
/* buf_table.c */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.hindex 1cf09e8f..e0bb4cc1 100644--- a/src/include/storage/bufmgr.h+++ b/src/include/storage/bufmgr.h@@ -410,6 +410,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,+ uint32 *complete_passes,+ uint32 *next_victim_buffer,+ uint64 *buffer_total_allocs,+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.plindex ae977297..f68e08e5 100644--- a/src/test/recovery/t/027_stream_regress.pl+++ b/src/test/recovery/t/027_stream_regress.pl@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.+# Increase to 32MB, so that we don't get "no unpinned buffers".+$node_primary->append_conf('postgresql.conf',+ 'shared_buffers = 32MB');+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 66809af9..b19f42d9 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -449,6 +449,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.50.1
[text/x-patch] v20260915a-0005-clock-sweep-balancing-of-allocations.patch (27.4K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/6-v20260915a-0005-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From 2238bee6341d6e491bca85a7f63843abfbb4693f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Tue, 2 Jun 2026 22:28:10 +0200
Subject: [PATCH v20260915a 5/9] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 428 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 471 insertions(+), 21 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sqlindex 92176fed..43d2e84f 100644--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql@@ -20,7 +20,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)+ num_allocs bigint, -- handled allocs (current cycle)+ total_req_allocs bigint, -- requested allocs (running total)+ num_req_allocs bigint, -- handled allocs (current cycle)+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.cindex b07fafda..f26b2332 100644--- a/contrib/pg_buffercache/pg_buffercache_pages.c+++ b/contrib/pg_buffercache/pg_buffercache_pages.c@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"+#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/tuplestore.h"
@@ -31,7 +33,7 @@
#define NUM_BUFFERCACHE_MARK_DIRTY_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 9+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -940,6 +942,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -971,6 +975,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",+ INT8OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",+ INT8OID, -1, 0);+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -992,11 +1002,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;+ uint64 buffer_total_allocs,+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;+ buffer_allocs,+ buffer_req_allocs;++ int *weights;+ Datum *dweights;+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -1005,8 +1021,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,- &buffer_total_allocs, &buffer_allocs);+ &complete_passes, &next_victim_buffer,+ &buffer_total_allocs, &buffer_allocs,+ &buffer_total_req_allocs, &buffer_req_allocs,+ &weights);++ dweights = palloc_array(Datum, funcctx->max_calls);+ for (int i = 0; i < funcctx->max_calls; i++)+ dweights[i] = Int32GetDatum(weights[i]);++ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -1035,6 +1059,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
+ values[9] = Int64GetDatum(buffer_total_req_allocs);+ nulls[9] = false;++ values[10] = Int64GetDatum(buffer_req_allocs);+ nulls[10] = false;++ values[11] = PointerGetDatum(array);+ nulls[11] = false;+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.cindex b58f59d5..a601c505 100644--- a/src/backend/storage/buffer/bufmgr.c+++ b/src/backend/storage/buffer/bufmgr.c@@ -4149,6 +4149,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */+ StrategySyncBalance();+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex 2d565796..a543fb12 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -35,6 +35,26 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*+ * XXX We need to make ClockSweep fixed-size, so that we can have an array+ * in shared memory. The easiest way is to pick a sufficiently high value+ * that no system will actually need. 32 seems high enough.+ *+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.+ */+#define MAX_BUFFER_PARTITIONS 32++/*+ * Coefficient used to combine the old and new balance coefficients, using+ * weighted average, so that we don't flap too much. The higher the value, the+ * more the old value affects the result.+ *+ * XXX Doesn't this obscure the interpretation of weights as probabilities to+ * allocate from a given partition? Does it still sum to 100%? I don't think+ * so, it's just a fraction of allocations to go from a given partition.+ */+#define CLOCKSWEEP_HISTORY_COEFF 0.5+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -68,9 +88,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*+ * Buffers that should have been allocated in this partition (but might+ * have been redirected to keep allocations balanced).+ */+ pg_atomic_uint32 numRequestedAllocs;+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;+ /*+ * Weights to balance buffer allocations for all the partitions. Each+ * partition gets a vector of weights 0-100, determining what fraction+ * of buffers to allocate from that partition. So [75, 15, 5, 5] would+ * mean 75% allocations should go from partition 0, 15% from partition+ * 1, and 5% from partitions 2&3. Each partition gets a different vector+ * of weights.+ *+ * Backends use the budget from it's "home" partition, so that a busy+ * partitions (with a lot of processes on that NUMA node etc.) spread+ * the allocations evenly.+ *+ * XXX Allocate a fixed-length array, to simplify working with array of+ * the structs, etc.+ */+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -140,7 +183,66 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint64 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);+static ClockSweep *ChooseClockSweep(bool balance);++/*+ * clocksweep allocation balancing+ *+ * To balance allocations from clocksweep partitions, each partition gets a+ * budget for allocating buffers from other partitions. A process that+ * "exhausts" a budget in it's home partition gets redirected to the other+ * partitions, driven by the budgets.+ *+ * For example, a partition may have budget [25, 25, 25, 25], which means+ * each of the 4 partitions should get 1/4 of allocations. Or the buget+ * can be [50, 50, 0, 0], which means all allocations will go to the first+ * two partitions (one of them being the "home" one);+ *+ * We could do that based on a random number generator, but for now we+ * simply treat the values as a budget, i.e. a number of allocations to+ * serve from other partitions, and move in round-robin way.+ *+ * This is very simple/cheap, and over many allocations it has the same+ * effect. For periods of low activity it may diverge, but that does not+ * matter much (we care about high-activity periods much more).+ *+ * We intentionally keep the "budget" fairly low, with the sum for a given+ * partition 100. That means we get to the same partition after only 100+ * allocations, keeping it more balanced. We can make the budgets higher+ * (say, to match the expected number of allocations, i.e. bout the average+ * number of allocations from the past interval). Or maybe configurable.+ *+ * XXX We should always start allocating from the "home" partition, i.e.+ * from from it, and only then redirect to other partitions.+ *+ * XXX It probably is not great all the processes from that "home"+ * partition are coordinated, and move to between partitions at about the+ * same time. Not sure what to do about this.+ *+ * XXX We should also prefer other partitions from the same NUMA node (if+ * there are some). Probably by setting the budgets.+ *+ * FIXME Explain at which point are the budgets recalculated, by which+ * process, and how that affects other processes allocating buffers.+ */++/*+ * The "optimal" clock-sweep partition. After a backend gets moved to a+ * different NUMA node, we restart the balancing so that it uses the+ * correct "budget" from the new home partition.+ */+static int clocksweep_partition_home = -1;++/*+ * The partition the backend is currently allocating from (either the+ * home one, or one of the redirected ones).+ */+static int clocksweep_partition_current = -1;++/*+ * The number of buffers to allocate from the current partition.+ */+static int clocksweep_partition_budget = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -152,7 +254,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -300,11 +402,68 @@ ClockSweepPartitionIndex(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)+ChooseClockSweep(bool balance)
{
+ /* What's the "optimal" partition for this backend? */
int index = ClockSweepPartitionIndex();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];++ /*+ * Was the process migrated to a different NUMA node? If the home partition+ * changed, we need to reset the budget and start over, so that we correctly+ * prefer "nearby" partitions etc.+ *+ * XXX Could this be a problem when processes move all the time? I don't+ * think so - if a process moves between many partitions, that alone will+ * spread the allocations over partitions. Similarly, if there are many+ * processes, that should make it even more even.+ */+ if (clocksweep_partition_home != index)+ {+ clocksweep_partition_home = index;+ clocksweep_partition_current = index;+ clocksweep_partition_budget = sweep->balance[index];+ }++ /* we should have a valid partition */+ Assert(clocksweep_partition_home != -1);+ Assert(clocksweep_partition_current != -1);+ Assert(clocksweep_partition_budget >= 0);++ /*+ * When balancing allocations, redirect the allocations to other partitions+ * according to the budgets. We move through partitions in a round-robin way,+ * after allocating the "budget" of allocations from the current one.+ */+ if (balance)+ {+ /*+ * Ran out of budget from the current partition? Move to the next one+ * with non-zero budget.+ */+ while (clocksweep_partition_budget == 0)+ {+ /* wrap around at the end */+ clocksweep_partition_current++;+ if (clocksweep_partition_current >= StrategyControl->num_partitions)+ clocksweep_partition_current = 0;++ clocksweep_partition_budget+ = sweep->balance[clocksweep_partition_current];+ }- return &StrategyControl->sweeps[index];+ /* account for the current allocation */+ --clocksweep_partition_budget;++ /*+ * Account for the allocation in the "home" partition, so that the next+ * round of rebalancing (recalculating the budgets) knows about the+ * allocation traffic in various partitions.+ */+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);+ }++ return &StrategyControl->sweeps[clocksweep_partition_current];
}
/*
@@ -381,7 +540,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* between CPUs / NUMA nodes in between, these call may pick different
* partitions, confusing the logic a bit.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -485,6 +644,229 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
}
}
+/*+ * StrategySyncBalance+ * update partition budgets, to balance the buffer allocations+ *+ * We want to give preference to allocating buffers on the same NUMA node,+ * but that might lead to imbalance - a single process would only use a+ * fraction of shared buffers. We don't want that, we want to utilize the+ * whole shared buffers. The number of allocations in each partition may+ * also change over time, so we need to adapt to that.+ *+ * To allow this "adaptive balancing", each partition has a set of weights,+ * determining what fraction of allocations to direct to other partitions.+ * For simplicity the coefficients are integers 0-100, expressing the+ * percentage of allocations redirected to that partition.+ *+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.+ * This means 50% of allocations will be redirected to partition 0, 25%+ * to partitions 1 and 2, and no allocations will go to partition 3.+ *+ * This means an allocation may be requested in partition A (i.e. the+ * home partition of the process requesting it), but end up allocating+ * the buffer in partition B. We have a counter for both - the number of+ * allocations requested in a partition, and the number of allocations+ * actually handled by that partition. The former is used for calculating+ * weights, the latter is used only for monitoring.+ *+ * The balancing happens in intervals - it adjusts future allocations+ * based on stats about recent allocations, namely:+ *+ * - numBufferAllocs - number of allocations served by a partition+ *+ * - numRequestedAllocs - number of allocatios requested in a partition+ *+ * We're trying to smooth numBufferAllocs in the next interval, based on+ * numRequestedAllocs measured in the last interval.+ *+ * The balancing algorithm works like this:+ *+ * - the target (average number of allocations per partition) is calculated+ * from total number of allocations requested in the last intervaal+ *+ * - partitions get divided into two groups - those with more allocation+ * requests than the target, and those with fewer requests+ *+ * - we "distribute" the delta (which is the same between the groups)+ * between the groups (one has more, the other fewer)+ *+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,+ * with each target allocation getting a proportional part (with respect+ * to the total delta).+ *+ * XXX Currently this does not give preference to other partitions on the+ * same NUMA node (redirect to it first), but it could.+ */+void+StrategySyncBalance(void)+{+ /* snapshot of allocation requests for partitions */+ uint32 allocs[MAX_BUFFER_PARTITIONS];++ uint32 total_allocs = 0, /* total number of allocations */+ avg_allocs, /* average allocations (per partition) */+ delta_allocs = 0; /* sum of allocs above average */++ /*+ * Collect the number of allocations requested in the past interval.+ * While at it, reset the counter to start the new interval.+ *+ * XXX We lock the partitions one by one, so this is not a perfectly+ * consistent snapshot of the counts, and the resets happen before we+ * update the weights too. But we're only looking for heuristics, so+ * this should be good enough.+ *+ * XXX A similar issue applies to the counter reset later - we haven't+ * updated the weights yet, so some of the requests counted for the next+ * interval will be redirected per current weights. Should be fine, it's+ * just an approximate heuristics, and there should be very few requests in+ * between. Alternatively, we could reset the request counters when setting+ * the new weights, and just ignore the couple requests in between.+ *+ * XXX Does this need to worry about the completePasses too?+ */+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ ClockSweep *sweep = &StrategyControl->sweeps[i];++ /* no need for a spinlock */+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);++ /* add the allocs to running total */+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);++ total_allocs += allocs[i];+ }++ /* Calculate the "fair share" of allocations per partition. */+ avg_allocs = (total_allocs / StrategyControl->num_partitions);++ /*+ * Calculate the "delta" from balanced state for each partition, i.e. how+ * many more/fewer allocations it handled relative to the average.+ */+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ if (allocs[i] > avg_allocs)+ delta_allocs += (allocs[i] - avg_allocs);+ }++ /*+ * Skip rebalancing when there's not enough activity, and just keep the+ * current weights.+ *+ * XXX The threshold of 100 allocation is pretty arbitrary.+ *+ * XXX Maybe a better strategy would be to slowly return to the default+ * weights, with each partition allocation only from itself?+ *+ * XXX Maybe we shouldn't even reset the counters in this case? But it+ * should not matter, if the activity is low.+ */+ if (avg_allocs < 100)+ {+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",+ avg_allocs);+ return;+ }++ /*+ * Likewise, skip rebalancing if the misbalance is not significant. We+ * consider it acceptable if the amount of allocations we'd need to+ * redistribute is less than 10% of the average.+ *+ * XXX Again, these threshold are rather arbitrary. And maybe we should+ * do the rabalancing in this case anyway, it's likely cheap and on a big+ * system 10% can be quite a lot.+ */+ if (delta_allocs < (avg_allocs * 0.1))+ {+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",+ delta_allocs, (uint32) (avg_allocs * 0.1));+ return;+ }++ /*+ * The actual rebalancing+ *+ * Partition with fewer than average allocations, should not redirect any+ * allocations to other partitions. So just use weights with a single+ * non-zero weight for the partition itself.+ *+ * Partition with more than average allocations, should not receive any+ * redirected allocations, and instead it should redirect excess allocations+ * to other partitions.+ *+ * The redistribution is "proportional" - if the excess allocations of a+ * partition represent 10% of the "delta", then each partition that+ * needs more allocations will get 10% of the gap from it.+ *+ * XXX We should add hysteresis, so that it does not oscillate or something+ * like that. Maybe CLOCKSWEEP_HISTORY_COEFF already does that?+ *+ * XXX Ideally, the alternative partitions to use first would be the other+ * partitions for the same node (if any).+ */+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ ClockSweep *sweep = &StrategyControl->sweeps[i];+ uint8 balance[MAX_BUFFER_PARTITIONS];++ /* lock, we're going to modify the balance weights */+ SpinLockAcquire(&sweep->clock_sweep_lock);++ /* reset the weights to start from scratch */+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);++ /* does this partition has fewer or more than avg_allocs? */+ if (allocs[i] < avg_allocs)+ {+ /* fewer - don't redirect any allocations elsewhere */+ balance[i] = 100;+ }+ else+ {+ /*+ * more - redistribute the excess allocations+ *+ * Each "target" partition (with less than avg_allocs) should get+ * a fraction proportional to (excess/delta) from this one.+ */++ /* fraction of the "total" delta */+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;++ /* keep just enough allocations to meet the target */+ balance[i] = (100.0 * avg_allocs / allocs[i]);++ /* redirect the extra allocations */+ for (int j = 0; j < StrategyControl->num_partitions; j++)+ {+ /* How many allocations to receive from i-th partition? */+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);++ /* ignore partitions that don't need additional allocations */+ if (allocs[j] > avg_allocs)+ continue;++ /* fraction to redirect */+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;+ }+ }++ /* combine the old and new weights (hysteresis) */+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)+ {+ sweep->balance[j]+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] ++ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];+ }++ SpinLockRelease(&sweep->clock_sweep_lock);+ }+}+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -657,7 +1039,21 @@ StrategyCtlShmemInit(void *arg)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);++ /*+ * Initialize the weights - start by allocating 100% buffers from+ * the current node / partition.+ */+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)+ {+ if (i == j)+ StrategyControl->sweeps[i].balance[i] = 100;+ else+ StrategyControl->sweeps[i].balance[j] = 0;+ }
}
/* No pending notification */
@@ -1025,8 +1421,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,- uint64 *buffer_total_allocs, uint32 *buffer_allocs)+ uint32 *complete_passes, uint32 *next_victim_buffer,+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1034,11 +1432,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
++ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);- /* calculate the actual buffer ID */- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);++ /* return the weights in a newly allocated array */+ *weights = palloc_array(int, StrategyControl->num_partitions);+ for (int i = 0; i < StrategyControl->num_partitions; i++)+ {+ (*weights)[i] = (int) sweep->balance[i];+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.hindex f9736c27..d39bd7e8 100644--- a/src/include/storage/buf_internals.h+++ b/src/include/storage/buf_internals.h@@ -600,6 +600,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.hindex e0bb4cc1..02833b19 100644--- a/src/include/storage/bufmgr.h+++ b/src/include/storage/bufmgr.h@@ -411,11 +411,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,- uint32 *next_victim_buffer,- uint64 *buffer_total_allocs,- uint32 *buffer_allocs);-+ uint32 *complete_passes,+ uint32 *next_victim_buffer,+ uint64 *buffer_total_allocs,+ uint32 *buffer_allocs,+ uint64 *buffer_total_req_allocs,+ uint32 *buffer_req_allocs,+ int **weights);
/* inline functions */
--
2.50.1
[text/x-patch] v20260915a-0006-clock-sweep-scan-all-partitions.patch (6.2K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/7-v20260915a-0006-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From b4ee4f6d4e91a44fe95dad794865975c2f96ee6f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@vondra.me>
Date: Tue, 2 Jun 2026 22:39:38 +0200
Subject: [PATCH v20260915a 6/9] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 83 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex a543fb12..1ac1e3e3 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -184,6 +184,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,+ BufferAccessStrategy strategy,+ uint64 *buf_state);
/*
* clocksweep allocation balancing
@@ -251,10 +254,9 @@ static int clocksweep_partition_budget = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -486,7 +488,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;+ ClockSweep *sweep,+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -545,33 +548,61 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at- * buffers from a single partition, aligned with the NUMA node. That means- * a process "sweeps" only a fraction of buffers, even if the other buffers- * are better candidates for eviction. Maybe there should be some logic to- * "steal" buffers from other partitions or other nodes?- *- * XXX This only searches a single partition, which can result in "no- * unpinned buffers available" even if there are buffers in other- * partitions. Needs to scan partitions if needed, as a fallback.- *- * XXX Would that also mean we should have multiple bgwriters, one for each- * node, or would one bgwriter still handle all nodes?- *- * XXX Also, the trycounter should not be set to NBuffers, but to buffer- * count for that one partition. In fact, this should not call ClockSweepTick- * for every iteration. The call is likely quite expensive (does a lot- * of stuff), and also may return a different partition on each call.- * We should just do it once, and then do the for(;;) loop. And then- * maybe advance to the next partition, until we scan through all of them.+ * Start with the "preferred" partition, and then proceed in a round-robin+ * manner. If we cycle back to the starting partition, it means none of the+ * partitions has unpinned buffers.
*/
- trycounter = NBuffers;+ sweep = ChooseClockSweep(true);+ sweep_start = sweep;+ for (;;)+ {+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);++ /* found a buffer in the "sweep" partition, we're done */+ if (buf != NULL)+ return buf;++ /*+ * Try advancing to the next partition, round-robin (if last partition,+ * wrap around to the beginning).+ *+ * XXX This is a bit ugly, there must be a better way to advance to the+ * next partition.+ */+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])+ sweep = StrategyControl->sweeps;+ else+ sweep++;++ /* we've scanned all partitions */+ if (sweep == sweep_start)+ break;+ }++ /* we shouldn't get here if there are unpinned buffers */+ elog(ERROR, "no unpinned buffers available");+}++/*+ * StrategyGetBufferPartition+ * get a free buffer from a single clock-sweep partition+ *+ * Returns NULL if there are no free (unpinned) buffers in the partition.+*/+static BufferDesc *+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,+ uint64 *buf_state)+{+ BufferDesc *buf;+ int trycounter;++ trycounter = sweep->numBuffers;
for (;;)
{
uint64 old_buf_state;
uint64 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -599,7 +630,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");+ return NULL;
}
break;
}
@@ -618,7 +649,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.plindex f68e08e5..ae977297 100644--- a/src/test/recovery/t/027_stream_regress.pl+++ b/src/test/recovery/t/027_stream_regress.pl@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.-# Increase to 32MB, so that we don't get "no unpinned buffers".-$node_primary->append_conf('postgresql.conf',- 'shared_buffers = 32MB');-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.50.1
[text/x-patch] v20260915a-0007-clock-sweep-cached-CPU-NUMA-node-and-loca.patch (5.2K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/8-v20260915a-0007-clock-sweep-cached-CPU-NUMA-node-and-loca.patch)
download | inline diff:
From 5e904a2f9e7041fcb7f89193da3fe8b7018beb70 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Thu, 10 Sep 2026 10:50:20 -0400
Subject: [PATCH v20260915a 7/9] clock-sweep: cached CPU/NUMA node and
locality-aware balancing (0008, Vondra/Wartak)
---
src/backend/storage/buffer/freelist.c | 84 +++++++++++++++++++++++----
1 file changed, 73 insertions(+), 11 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex 1ac1e3e3..8e3f7c31 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -55,6 +55,9 @@
*/
#define CLOCKSWEEP_HISTORY_COEFF 0.5
+/* How often backend should re-fetch the CPU/node on which it is running on? */+#define CLOCKSWEEP_CPU_NODE_REFRESH 128+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -348,13 +351,29 @@ ClockSweepPartitionIndex(void)
#ifdef USE_LIBNUMA
if (shared_buffers_numa)
{
- int cpu;+ /*+ * Cache the CPU/NUMA node, refreshing only every CLOCKSWEEP_CPU_NODE_REFRESH+ * allocations. It appears that sched_getcpu()/numa_node_of_cpu() are not free.+ * On some platforms it take price of full system call, or the rest (x86_64?)+ * is can be use VDSO optimization. The backend rarely migrates between NUMA+ * nodes, and the balance logic only needs to notice migration after some time,+ * so an occasional refresh is good enough.+ */+ static int cached_node = -1;+ static uint32 refresh_counter = 0;- /* XXX do we need to check sched_getcpu is available, somehow? */- if ((cpu = sched_getcpu()) < 0)+ if (cached_node < 0 || (refresh_counter++ % CLOCKSWEEP_CPU_NODE_REFRESH) == 0)+ {+ int cpu;++ /* XXX do we need to check sched_getcpu is available, somehow? */+ if ((cpu = sched_getcpu()) < 0)
elog(ERROR, "sched_getcpu failed: %m");
- node = numa_node_of_cpu(cpu);+ /* XXX/JW: use libnuma wrapper for this */+ cached_node = numa_node_of_cpu(cpu);+ }+ node = cached_node;
}
#endif
@@ -737,7 +756,8 @@ StrategySyncBalance(void)
uint32 total_allocs = 0, /* total number of allocations */
avg_allocs, /* average allocations (per partition) */
- delta_allocs = 0; /* sum of allocs above average */+ delta_allocs = 0, /* sum of allocs above average */+ redirect_cutoff; /* redirect only above this many allocs */
/*
* Collect the number of allocations requested in the past interval.
@@ -818,6 +838,20 @@ StrategySyncBalance(void)
return;
}
+ /*+ * A partition only redirects allocations to other partitions when it+ * exceeds the average by more than some threshold percent.+ * Below this cutoff we keep allocations local, to preserve NUMA locality.+ *+ * TODO: maybe better value is possible. On 4s with 25 I've got good results,+ * but with value of 50 I've got slight degradation. Maybe it should + * be equal to 100/numa_nodes ?+ *+ */+#define CLOCKSWEEP_CUTOFF_THRESHOLD 25+ redirect_cutoff = avg_allocs ++ (uint32) ((uint64) avg_allocs * CLOCKSWEEP_CUTOFF_THRESHOLD / 100);+
/*
* The actual rebalancing
*
@@ -850,10 +884,15 @@ StrategySyncBalance(void)
/* reset the weights to start from scratch */
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
- /* does this partition has fewer or more than avg_allocs? */- if (allocs[i] < avg_allocs)+ /*+ * Does this partition exceed its fair share by more than the+ * threshold? If not, keep all allocations local - redirecting them+ * would push memory onto remote NUMA nodes for no real benefit when+ * the load is already close to balanced.+ */+ if (allocs[i] <= redirect_cutoff)
{
- /* fewer - don't redirect any allocations elsewhere */+ /* near fair share (or below) - keep allocations local */
balance[i] = 100;
}
else
@@ -868,22 +907,45 @@ StrategySyncBalance(void)
/* fraction of the "total" delta */
double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
- /* keep just enough allocations to meet the target */- balance[i] = (100.0 * avg_allocs / allocs[i]);+ /* how much we keep local; we hand out the rest below */+ int kept = 100;
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
/* How many allocations to receive from i-th partition? */
uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ int w;++ /* do not redirect to ourselves */+ if (j == i)+ continue;
/* ignore partitions that don't need additional allocations */
if (allocs[j] > avg_allocs)
continue;
+ /*+ * Only use other partitions that actually have demand of+ * their own (avoid idle). If we fail, there's always the+ * scan-all-partitions fallback.+ *+ * TODO:: just guessing,heuristics+ */+ if (allocs[j] < (avg_allocs / 2))+ continue;+
/* fraction to redirect */
- balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;+ w = (int) ((100.0 * receive_allocs / allocs[i]) + 0.5);+ balance[j] = w;+ kept -= w;
}
++ /* avoid negative balances */+ if (kept > 0)+ balance[i] = kept;+ else+ balance[i] = 1;
}
/* combine the old and new weights (hysteresis) */
--
2.50.1
[text/x-patch] v20260915a-0008-pg_buffercache-install-the-1.7-1.8-upgrad.patch (798B, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/9-v20260915a-0008-pg_buffercache-install-the-1.7-1.8-upgrad.patch)
download | inline diff:
From a3cd85627f44c1db36672827657a8994b1663c87 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Thu, 10 Sep 2026 11:44:49 -0400
Subject: [PATCH v20260915a 8/9] pg_buffercache: install the 1.7--1.8 upgrade
script in meson build (patchset updated Makefile only)
---
contrib/pg_buffercache/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/contrib/pg_buffercache/meson.build b/contrib/pg_buffercache/meson.buildindex e681205a..361628b8 100644--- a/contrib/pg_buffercache/meson.build+++ b/contrib/pg_buffercache/meson.build@@ -25,6 +25,7 @@ install_data(
'pg_buffercache--1.4--1.5.sql',
'pg_buffercache--1.5--1.6.sql',
'pg_buffercache--1.6--1.7.sql',
+ 'pg_buffercache--1.7--1.8.sql',
'pg_buffercache.control',
kwargs: contrib_data_args,
)
--
2.50.1
[text/x-patch] v20260915a-0009-Replace-the-usage_count-clock-sweep-with-.patch (25.6K, ../dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com/10-v20260915a-0009-Replace-the-usage_count-clock-sweep-with-.patch)
download | inline diff:
From 19367173122b0fb857a31be9757847c3a3e8374b Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Mon, 6 Jul 2026 19:22:45 -0400
Subject: [PATCH v20260915a 9/9] Replace the usage_count clock sweep with a
cooling-stage evictor
Replace the 0..5 usage_count buffer-replacement policy with a cooling-stage
clock (the LeanStore / 2Q-A1 model): a buffer is simply HOT (recently used)
or COOL (an eviction candidate), with "pinned" being the existing refcount.
There is no per-buffer access counter.
- A demand-loaded page is admitted COOL (probationary), not HOT. A second
access via PinBuffer promotes it COOL -> HOT (the rescue). So a page
touched once -- a sequential scan -- fills and drains the COOL stage and
is evicted from it without ever displacing the HOT working set. Scan
resistance is intrinsic to the replacement algorithm, which is what lets a
later commit remove the BufferAccessStrategy ring buffers entirely.
- The foreground sweep in StrategyGetBuffer() reclaims an already-COOL,
unpinned buffer, pinning it with a CAS so a racing PinBuffer always wins.
Promotion (COOL -> HOT) and demotion (HOT -> COOL) are single-bit
transitions; only the eviction claim is a CAS.
- The background writer maintains the supply of COOL victims. As its LRU
scan runs ahead of the clock hand it demotes HOT buffers to COOL, so the
foreground finds a victim in a single pass rather than having to cool
buffers itself. The demotion is demand-driven -- bounded by the predicted
allocation for the next cycle -- so it stages just enough COOL buffers
without cooling the whole pool, and it is done under the buffer header
lock the scan already holds. A single second-chance reference bit
(set by PinBuffer, cleared on the bgwriter's first pass over a HOT buffer)
spares a recently-accessed buffer one cooling pass, keeping the genuinely
hot set out of the COOL stage under scan pressure. BgBufferSync also
tracks the reusable-buffer density it directly observes on a shorter
smoothing window, so a burst of probationary/scan COOL pages is followed
promptly rather than averaged away. Under a bulk-dirtying workload the
per-cycle clean-write cap is raised from bgwriter_lru_maxpages to predicted
demand so the bgwriter keeps supplying clean victims, rather than the
foreground sweep having to flush dirty victims inline; normal workloads,
where demand is below the cap, are unaffected.
The 4-bit usage_count field is reinterpreted in place: bit 0 is the HOT/COOL
state (BUF_COOLSTATE_ONE), bit 1 the reference bit (BUF_REFBIT). The 64-bit
buffer-state layout -- refcount, flag and lock offsets and their StaticAsserts
-- is unchanged; only the meaning of the field and the instructions that touch
it change. A StaticAssert requires the field to be at least 2 bits wide so a
future width change cannot push the reference bit into the flag bits.
BM_MAX_USAGE_COUNT becomes BUF_COOLSTATE_HOT (1), so the pin fast path
saturates at HOT. Local (temp-table) buffers get the same two-state
treatment; being single-backend they need no background cooler.
Because the reference bit shares the field, the full 4-bit value can be 0..3;
readers that mean "the cooling state" must use BUF_STATE_GET_COOLSTATE(), which
masks to bit 0 and returns only 0 (COOL) or 1 (HOT), never the raw field.
contrib/pg_buffercache is updated accordingly: its usagecount column and the
pg_buffercache_summary average report the cooling state (0 = COOL, 1 = HOT),
and pg_buffercache_usage_counts() buckets on it. Using the raw 4-bit getter
there would index its BM_MAX_USAGE_COUNT+1 = 2-element arrays with values up
to 3 and overrun the stack; masking to the cooling bit keeps the index in
range. The reference bit is deliberately not exposed as usagecount.
Depends on the batched clock sweep from the previous commit.
---
contrib/pg_buffercache/pg_buffercache_pages.c | 6 +-
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++----
src/backend/storage/buffer/freelist.c | 54 +++++---
src/backend/storage/buffer/localbuf.c | 13 +-
src/include/storage/buf_internals.h | 63 +++++++--
5 files changed, 205 insertions(+), 59 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.cindex f26b2332..56fbac49 100644--- a/contrib/pg_buffercache/pg_buffercache_pages.c+++ b/contrib/pg_buffercache/pg_buffercache_pages.c@@ -165,7 +165,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS)
reldatabase = bufHdr->tag.dbOid;
forknum = BufTagGetForkNum(&bufHdr->tag);
blocknum = bufHdr->tag.blockNum;
- usagecount = BUF_STATE_GET_USAGECOUNT(buf_state);+ usagecount = BUF_STATE_GET_COOLSTATE(buf_state);
pinning_backends = BUF_STATE_GET_REFCOUNT(buf_state);
if (buf_state & BM_DIRTY)
@@ -609,7 +609,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS)
if (buf_state & BM_VALID)
{
buffers_used++;
- usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state);+ usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state);
if (buf_state & BM_DIRTY)
buffers_dirty++;
@@ -659,7 +659,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS)
CHECK_FOR_INTERRUPTS();
- usage_count = BUF_STATE_GET_USAGECOUNT(buf_state);+ usage_count = BUF_STATE_GET_COOLSTATE(buf_state);
usage_counts[usage_count]++;
if (buf_state & BM_DIRTY)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.cindex a601c505..878285b0 100644--- a/src/backend/storage/buffer/bufmgr.c+++ b/src/backend/storage/buffer/bufmgr.c@@ -83,6 +83,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_COOLED 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -634,7 +635,7 @@ static void PinBuffer_Locked(BufferDesc *buf);
static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,+static int SyncOneBuffer(int buf_id, bool skip_recently_used, bool cool_if_hot,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
@@ -2333,7 +2334,10 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* checkpoints, except for their "init" forks, which need to be treated
* just like permanent relations.
*/
- set_bits |= BM_TAG_VALID | BUF_USAGECOUNT_ONE;+ set_bits |= BM_TAG_VALID;+ /* Admit the newly loaded page COOL (probation); a second access via+ * PinBuffer promotes it to HOT. This is what makes a one-touch scan+ * self-evicting -- see the cooling-state notes in buf_internals.h. */
if (relpersistence == RELPERSISTENCE_PERMANENT || forkNum == INIT_FORKNUM)
set_bits |= BM_PERMANENT;
@@ -3002,7 +3006,9 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
victim_buf_hdr->tag = tag;
- set_bits |= BM_TAG_VALID | BUF_USAGECOUNT_ONE;+ set_bits |= BM_TAG_VALID;+ /* Admit COOL (probation); see the comment at the other admission+ * site and the cooling-state notes in buf_internals.h. */
if (bmr.relpersistence == RELPERSISTENCE_PERMANENT || fork == INIT_FORKNUM)
set_bits |= BM_PERMANENT;
@@ -3332,21 +3338,17 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy,
/* increase refcount */
buf_state += BUF_REFCOUNT_ONE;
- if (strategy == NULL)- {- /* Default case: increase usagecount unless already max. */- if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)- buf_state += BUF_USAGECOUNT_ONE;- }- else- {- /*- * Ring buffers shouldn't evict others from pool. Thus we- * don't make usagecount more than 1.- */- if (BUF_STATE_GET_USAGECOUNT(buf_state) == 0)- buf_state += BUF_USAGECOUNT_ONE;- }+ /*+ * Accessing a resident buffer promotes it to HOT (the 2Q rescue):+ * a page loaded COOL on probation becomes part of the hot working+ * set on its second touch. BM_MAX_USAGE_COUNT is+ * BUF_COOLSTATE_HOT (1), so this saturates at HOT and never+ * overflows the field. We also set the second-chance ref bit so+ * the bgwriter's next cooling pass spares this recently-used buffer.+ */+ if (BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT)+ buf_state += BUF_COOLSTATE_ONE;+ buf_state |= BUF_REFBIT;
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
buf_state))
@@ -3785,7 +3787,7 @@ BufferSync(int flags)
*/
if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)+ if (SyncOneBuffer(buf_id, false, false, &wb_context) & BUF_WRITTEN)
{
TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
PendingCheckpointerStats.buffers_written++;
@@ -3877,6 +3879,19 @@ BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
float smoothing_samples = 16;
float scan_whole_pool_milliseconds = 120000.0;
+ /*+ * The cleaner scan directly observes the reusable (COOL, unpinned) buffer+ * density over the region it walks, which -- with the cooling-stage+ * evictor -- is exactly the sweep's victim predicate. That observation is+ * ground truth for the buffers about to be reused, whereas the strategy+ * scan's positional proxy (strategy_delta/recent_alloc) blurs a pool whose+ * COOL population is spatially clustered (a scan burst leaves whole regions+ * COOL, hot OLTP regions not). So we let the cleaner's own sample adapt on+ * a shorter window than the strategy proxy, tracking a burst of+ * probationary/scan COOL pages within a cycle or two instead of lagging it.+ */+ float cleaner_smoothing_samples = 4;+
/* Used to compute how far we scan ahead */
long strategy_delta;
int bufs_to_lap;
@@ -3890,6 +3905,7 @@ BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
int num_to_scan;
int num_written;
int reusable_buffers;
+ int write_limit;
/* Variables for final smoothed_density update */
long new_strategy_delta;
@@ -4056,8 +4072,22 @@ BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
* Now write out dirty reusable buffers, working forward from the
* next_to_clean point, until we have lapped the strategy scan, or cleaned
* enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.- */+ * requirements, or hit the write limit.+ *+ * The per-cycle write cap is normally bgwriter_lru_maxpages. But under a+ * bulk-dirtying workload (COPY, bulk UPDATE, VACUUM) the pool fills with+ * dirty COOL buffers faster than that fixed cap can clean, so the+ * foreground clock sweep is forced to flush dirty victims inline -- the+ * very cost the cooling-stage evictor is meant to keep off the critical+ * path. When predicted demand (upcoming_alloc_est) exceeds the fixed cap,+ * raise the limit to meet demand so the bgwriter stays ahead and supplies+ * clean victims. This stays bounded (by demand and by lapping the+ * strategy point), so it cannot run away, and normal workloads are+ * unaffected because there upcoming_alloc_est <= bgwriter_lru_maxpages.+ */+ write_limit = bgwriter_lru_maxpages;+ if (upcoming_alloc_est > write_limit)+ write_limit = upcoming_alloc_est;
num_to_scan = bufs_to_lap;
num_written = 0;
@@ -4066,7 +4096,13 @@ BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(saved->next_to_clean, true,+ /*+ * cool_if_hot = false: with the cool-in-place foreground sweep, the+ * clock hand demotes HOT buffers on every tick, so the bgwriter does+ * writeback only and leaves all cooling to the sweep (no background+ * pre-cooling, no double-cooling).+ */+ int sync_state = SyncOneBuffer(saved->next_to_clean, true, false,
wb_context);
if (++saved->next_to_clean >= (first_buffer + num_buffers))
@@ -4114,7 +4150,7 @@ BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
{
scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;+ cleaner_smoothing_samples;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
@@ -4211,16 +4247,26 @@ BgBufferSync(WritebackContext *wb_context)
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
*
+ * If cool_if_hot is true (the bgwriter's LRU scan), an unpinned HOT buffer is+ * demoted HOT -> COOL as we pass it, pre-staging eviction candidates so the+ * foreground clock sweep finds a COOL victim in a single pass instead of+ * having to cool buffers itself (force_cool). The demotion is done under the+ * buffer header lock we already hold, so it needs no CAS and cannot race a+ * concurrent demotion. A concurrent PinBuffer promotes it back to HOT, which+ * is the intended 2Q behavior (a re-accessed buffer is rescued).+ *
* Returns a bitmask containing the following flag bits:
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
- * pin count 0 and usage count 0.+ * pin count 0 and is COOL (an eviction candidate).+ * BUF_COOLED: we demoted this buffer HOT -> COOL this call.
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)+SyncOneBuffer(int buf_id, bool skip_recently_used, bool cool_if_hot,+ WritebackContext *wb_context)
{
BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
int result = 0;
@@ -4242,8 +4288,38 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
*/
buf_state = LockBufHdr(bufHdr);
+ /*+ * Pre-cool with a second chance: if asked, act on an unpinned HOT buffer.+ * If its ref bit is set (accessed since our last pass), clear the ref bit+ * and leave it HOT -- a recently-used buffer earns one reprieve, keeping+ * the hot working set out of the COOL stage under scan pressure. Only a+ * HOT buffer whose ref bit is already clear is demoted HOT -> COOL,+ * pre-staging it as an eviction candidate for the foreground sweep. We+ * hold the header lock, so each transition is a plain masked store applied+ * atomically by UnlockBufHdrExt; we re-lock to continue the dirty-write+ * inspection below.+ */+ if (cool_if_hot &&+ BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&+ BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL)+ {+ if (BUF_STATE_GET_REFBIT(buf_state))+ {+ /* second chance: consume the ref bit, stay HOT */+ UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_REFBIT, 0);+ buf_state = LockBufHdr(bufHdr);+ }+ else+ {+ /* not re-accessed since last pass: demote to COOL */+ UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_USAGECOUNT_MASK, 0);+ buf_state = LockBufHdr(bufHdr);+ result |= BUF_COOLED;+ }+ }+
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
- BUF_STATE_GET_USAGECOUNT(buf_state) == 0)+ BUF_STATE_GET_COOLSTATE(buf_state) == BUF_COOLSTATE_COOL)
{
result |= BUF_REUSABLE;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.cindex 8e3f7c31..12ad0d4f 100644--- a/src/backend/storage/buffer/freelist.c+++ b/src/backend/storage/buffer/freelist.c@@ -620,6 +620,7 @@ StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
{
uint64 old_buf_state;
uint64 local_buf_state;
+ bool no_progress = false;
buf = GetBufferDescriptor(ClockSweepTick(sweep));
@@ -632,12 +633,7 @@ StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
{
local_buf_state = old_buf_state;
- /*- * If the buffer is pinned or has a nonzero usage_count, we cannot- * use it; decrement the usage_count (unless pinned) and keep- * scanning.- */-+ /* If the buffer is pinned we cannot use it; keep scanning. */
if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0)
{
if (--trycounter == 0)
@@ -661,9 +657,23 @@ StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
continue;
}
- if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)+ if (BUF_STATE_GET_COOLSTATE(local_buf_state) != BUF_COOLSTATE_COOL)
{
- local_buf_state -= BUF_USAGECOUNT_ONE;+ /*+ * HOT buffer: cool it in place this tick. Apply a single+ * second-chance reference bit: a HOT buffer whose ref bit is+ * set (touched since it was last passed) has the ref bit+ * cleared and stays HOT; only a HOT buffer whose ref bit is+ * already clear is demoted HOT -> COOL. Either transition is+ * progress toward a victim, so reset trycounter. We do NOT+ * claim the buffer this tick -- a demoted buffer becomes a+ * candidate for a later tick, giving it one more full sweep of+ * grace before eviction.+ */+ if (BUF_STATE_GET_REFBIT(local_buf_state))+ local_buf_state &= ~BUF_REFBIT; /* second chance: clear ref, stay HOT */+ else+ local_buf_state &= ~BUF_USAGECOUNT_MASK; /* HOT -> COOL, clear ref bit */
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
@@ -674,7 +684,7 @@ StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
}
else
{
- /* pin the buffer if the CAS succeeds */+ /* COOL and unpinned: claim it. Pin if the CAS succeeds. */
local_buf_state += BUF_REFCOUNT_ONE;
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
@@ -691,6 +701,15 @@ StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
}
}
}
++ /*+ * Only a pinned buffer is no progress. A full NBuffers pass that+ * makes no progress means every buffer is pinned, so fail rather than+ * spin forever. (A failed CAS above is neither progress nor a full+ * miss: we simply retry the same buffer.)+ */+ if (no_progress && --trycounter == 0)+ elog(ERROR, "no unpinned buffers available");
}
}
@@ -1398,14 +1417,17 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state)
/*
* If the buffer is pinned we cannot use it under any circumstances.
*
- * If usage_count is 0 or 1 then the buffer is fair game (we expect 1,- * since our own previous usage of the ring element would have left it- * there, but it might've been decremented by clock-sweep since then).- * A higher usage_count indicates someone else has touched the buffer,- * so we shouldn't re-use it.+ * If it is unpinned but has been promoted to HOT, another backend+ * touched it since we last cycled past this ring slot, so it has+ * joined the working set and we must not recycle it -- tell the caller+ * to get a fresh victim from the sweep instead. This is the 1-bit+ * cooling-state analog of the stock ring's "usage_count > 1 means+ * someone else touched it" test: a slot the ring keeps reusing and+ * nobody else pins stays COOL, and an out-of-ring PinBuffer() is+ * exactly what promotes it to HOT.
*/
- if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0- || BUF_STATE_GET_USAGECOUNT(local_buf_state) > 1)+ if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0 ||+ BUF_STATE_GET_COOLSTATE(local_buf_state) != BUF_COOLSTATE_COOL)
break;
/* See equivalent code in PinBuffer() */
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.cindex 4870c8e1..21c08091 100644--- a/src/backend/storage/buffer/localbuf.c+++ b/src/backend/storage/buffer/localbuf.c@@ -167,7 +167,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state = pg_atomic_read_u64(&bufHdr->state);
buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK);
- buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE;+ buf_state |= BM_TAG_VALID; /* admit COOL (probation) */
pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state);
*foundPtr = false;
@@ -248,9 +248,10 @@ GetLocalVictimBuffer(void)
{
uint64 buf_state = pg_atomic_read_u64(&bufHdr->state);
- if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0)+ if (BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL)
{
- buf_state -= BUF_USAGECOUNT_ONE;+ /* HOT: give it a second chance, cool it and keep scanning. */+ buf_state -= BUF_COOLSTATE_ONE;
pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
@@ -454,7 +455,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
victim_buf_hdr->tag = tag;
- buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE;+ buf_state |= BM_TAG_VALID; /* admit COOL (probation) */
pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state);
@@ -839,9 +840,9 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
NLocalPinnedBuffers++;
buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
- BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)+ BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT)
{
- buf_state += BUF_USAGECOUNT_ONE;+ buf_state += BUF_COOLSTATE_ONE;
}
pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.hindex d39bd7e8..064e547a 100644--- a/src/include/storage/buf_internals.h+++ b/src/include/storage/buf_internals.h@@ -67,6 +67,50 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_L
#define BUF_USAGECOUNT_ONE \
(UINT64CONST(1) << BUF_REFCOUNT_BITS)
+/*+ * Cooling state (LeanStore / 2Q-A1 cooling-stage clock sweep).+ *+ * The field historically used for the 0..5 usage_count now holds a single+ * cooling-state bit: HOT (recently accessed, not an eviction candidate) or+ * COOL (an eviction candidate). We reuse BUF_USAGECOUNT_ONE as the unit so+ * the buffer-state bit geography -- refcount, flag, and lock offsets, and the+ * 64-bit StaticAsserts -- is unchanged; only the meaning of the field and the+ * instructions that touch it change.+ *+ * A demand-loaded page is admitted COOL (probation); a second access promotes+ * it to HOT (the rescue). The sweep prefers COOL victims and demotes HOT to+ * COOL only when a full pass finds no COOL victim. So a one-touch scan fills+ * and drains the COOL stage without displacing the HOT working set -- scan+ * resistance intrinsic to the replacement algorithm.+ */+#define BUF_COOLSTATE_COOL 0+#define BUF_COOLSTATE_HOT 1+#define BUF_COOLSTATE_ONE BUF_USAGECOUNT_ONE++/*+ * Second-chance reference bit, bit 1 of the (former usagecount) field, one+ * position above the cooling-state bit. PinBuffer sets it on every access.+ * The bgwriter's pre-cooling gives a HOT buffer a second chance: the first+ * time it passes a HOT buffer whose ref bit is set, it clears the ref bit and+ * leaves the buffer HOT; only a HOT buffer whose ref bit is already clear (not+ * re-accessed since the previous bgwriter pass) is demoted to COOL. This+ * keeps genuinely-hot pages out of the COOL stage (protecting the working set+ * from being cooled under scan pressure) while leaving the foreground sweep a+ * single-pass search over the pre-staged COOL buffers. A separate bit (not a+ * count) so it stays a plain masked store under the header lock.+ */+#define BUF_REFBIT (UINT64CONST(2) << BUF_REFCOUNT_BITS)+#define BUF_STATE_GET_REFBIT(state) (((state) & BUF_REFBIT) != 0)++/*+ * The cooling state occupies bit 0 and the reference bit occupies bit 1 of+ * the (former usagecount) field, so the field must be at least 2 bits wide.+ * Assert it here so a future change to BUF_USAGECOUNT_BITS cannot silently+ * push BUF_REFBIT up into the flag bits.+ */+StaticAssertDecl(BUF_USAGECOUNT_BITS >= 2,+ "cooling state + reference bit need at least 2 bits in the usagecount field");+
/* flags related definitions */
#define BUF_FLAG_SHIFT \
(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)
@@ -92,6 +136,11 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_L
#define BUF_STATE_GET_USAGECOUNT(state) \
((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT))
+/* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so+ * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */+#define BUF_STATE_GET_COOLSTATE(state) \+ ((uint32) (((state) >> BUF_USAGECOUNT_SHIFT) & 1))+
/*
* Flags for buffer descriptors
*
@@ -134,17 +183,15 @@ StaticAssertDecl(MAX_BACKENDS_BITS <= (BUF_LOCK_BITS - 2),
/*
- * The maximum allowed value of usage_count represents a tradeoff between- * accuracy and speed of the clock-sweep buffer management algorithm. A- * large value (comparable to NBuffers) would approximate LRU semantics.- * But it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the- * clock-sweep hand to find a free buffer, so in practice we don't want the- * value to be very large.+ * The cooling state is a single bit (HOT/COOL); the maximum value stored in+ * the field is therefore BUF_COOLSTATE_HOT. Retained under the historical+ * name BM_MAX_USAGE_COUNT so the pin fast path ("promote unless already at+ * max") reads naturally.
*/
-#define BM_MAX_USAGE_COUNT 5+#define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT
StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS),
- "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits");+ "cooling state doesn't fit in BUF_USAGECOUNT_BITS bits");
/*
* Buffer tag identifies which disk block the buffer contains.
--
2.50.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@burd.me, pgsql-hackers@lists.postgresql.org, andres@anarazel.de, a.makhmutov@postgrespro.ru, tomas@vondra.me, jakub.wartak@enterprisedb.com
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <dba6522b-7de9-403f-80f5-783cb90d4647@app.fastmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox