public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Jakub Wartak <[email protected]>
Cc: Alexey Makhmutov <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: Andres Freund <[email protected]>
Subject: Re: Adding basic NUMA awareness
Date: Wed, 26 Nov 2025 17:19:12 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com>
References: <CAKZiRmy7VW5b0OKwHXtRSudCBFB48-kYnOEXeqrLfbTQ=5qQrA@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CAKZiRmzQ=jzWSz7NjHggyqpnMkZUaeO00t7rUovs+zvi_YY48w@mail.gmail.com>
<[email protected]>
<CAKZiRmx=0C5k3Qs0DdHZw9cL+72sX_ZH_RXdUW-7U1-978Kvnw@mail.gmail.com>
<[email protected]>
<CAKZiRmww2P6QAzu6W+vxB89i5Ha-YRSHMeyr6ax2Lymcu3LUcw@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com>
On 11/25/25 15:12, Jakub Wartak wrote:
> Hi Tomas!
>
> [..]
>> Which I think is mostly the same thing you're saying, and you have the maps to support it.
>
> Right, the thread is kind of long, you were right back then, well but
> at least we've got a solid explanation with data.
>
>> Here's an updated version of the patch series.
>
> Just for double confirmation, I've used those ones (v20251121*) and
> they indeed interleaved parts of shm memory.
>
>> It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
>> and a incorrect array length.
>
> You'll need to rebase again, pg_buffercache_numa got updated again on
> Monday and clashes with 0006.
>
Rebased patch series attached.
>> The main change is in 0006 - it sets the default allocation policy for
>> shmem to interleaving, before doing the explicit partitioning for shared
>> buffers. It does it by calling numa_set_membind before the mmap(), and
>> then numa_interleave_memory() on the allocated shmem. It does this to
>> allow using MAP_POPULATE - but that's commented out by default.
>>
>> This does seem to solve the SIGBUS failures for me. I still think there
>> might be a small chance of hitting that, because of locating an extra
>> "boundary" page on one of the nodes. But it should be solvable by
>> reserving a couple more pages.
>
> I can confirm, never got any SIGBUS during the later described
> benchmarks, so it's much better now.
>
Good!
>> Jakub, what do you think?
>
> On one side not using MAP_POPULATE gives instant startup, but on the
> other it gives much better predictability latencies especially fresh
> after starting up (this might matter to folks who like to benchmark --
> us?, but initially I've just used it as a simple hack to touch
> memory). I would be wary of using MAP_POPULATE with s_b when it would
> be sized in hundreths of GBs, it could take minutes in startup, which
> would be terrible if someone would hit SIGSEGV on production and
> expect restart_after_crash=true to save him. I mean WAL redo crash
> would be terrible, but that would be terrible * 2. Also pretty
> long-term with DIO, we'll get much bigger s_b anyway (hopefully), so
> it would hurt even more, so I think that would be a bad path(?)
>
I think the MAP_POPULATE should be optional, enabled by GUC.
> I've benchmarked the thing in two scenarios (readonly pgbench < s_b
> size across variations of code and connections and 2nd one with
> seqconcurrrentscans) in solid stable conditions: 4s32c64t == 4 NUMA
> nodes, 128GB RAM, 31GB shared_buffers dbsize ~29GB, 6.14.x, no idle
> CPU states, no turbo boost, and so on, literally great home heater
> when there's -3C outside!)
>
> The data is baseline "100%" for master along with HP on/off (so it's
> showing diff % from respective HP setting):
>
> scenario I: pgbench -S
>
> connections
> branch HP 1 8 64 128 1024
> master off 100.00% 100.00% 100.00% 100.00% 100.00%
> master on 100.00% 100.00% 100.00% 100.00% 100.00%
> numa16 off 99.13% 100.46% 99.66% 99.44% 89.60%
> numa16 on 101.80% 100.89% 99.36% 99.89% 93.43%
> numa4 off 96.82% 100.61% 99.37% 99.92% 94.41%
> numa4 on 101.83% 100.61% 99.35% 99.69% 101.48%
> pgproc16 off 99.13% 100.84% 99.38% 99.85% 91.15%
> pgproc16 on 101.72% 101.40% 99.72% 100.14% 95.20%
> pgproc4 off 98.63% 101.44% 100.05% 100.14% 90.97%
> pgproc4 on 101.05% 101.46% 99.92% 100.31% 97.60%
> sweep16 off 99.53% 101.14% 100.71% 100.75% 101.52%
> sweep16 on 97.63% 102.49% 100.42% 100.75% 105.56%
> sweep4 off 99.43% 101.59% 100.06% 100.45% 104.63%
> sweep4 on 97.69% 101.59% 100.70% 100.69% 104.70%
>
> I would consider everything +/- 3% as noise (technically each branch
> was a different compilation/ELF binary, as changing this #define
> required to do so to get 4 vs 16; please see attached script). I miss
> the explanation why without HP it deteriorates so much with for c=1024
> with the patches.
I wouldn't expect a big difference for "pgbench -S". That workload has
so much other fairly expensive stuff (e.g. initializing index scans
etc.), the cost of buffer replacement is going to be fairly limited.
The regressions for numa/pgproc patches with 1024 clients are annoying,
but how realistic is such scenario? With 32/64 CPUs, having 1024 active
connections is a substantial overload. If we can fix this, great. But I
think such regression may be OK if we get benefits for reasonable setups
(with fewer clients).
I don't know why it's happening, though. I haven't been testing cases
with so many clients (compared to number of CPUs).
>
> scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
> pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
> without PQ by:
> \set num (:client_id % 8) + 1
> select sum(octet_length(filler)) from pgbench_accounts_:num;
>
> connections
> branch HP 1 8 64 128
> master off 100.00% 100.00% 100.00% 100.00%
> master on 100.00% 100.00% 100.00% 100.00%
> numa16 off 115.62% 108.87% 101.08% 111.56%
> numa16 on 107.68% 104.90% 102.98% 105.51%
> numa4 off 113.55% 111.41% 101.45% 113.10%
> numa4 on 107.90% 106.60% 103.68% 106.98%
> pgproc16 off 111.70% 108.27% 98.69% 109.36%
> pgproc16 on 106.98% 100.69% 101.98% 103.42%
> pgproc4 off 112.41% 106.15% 100.03% 112.03%
> pgproc4 on 106.73% 105.77% 103.74% 101.13%
> sweep16 off 100.63% 100.38% 98.41% 103.46%
> sweep16 on 109.03% 99.15% 101.17% 99.19%
> sweep4 off 102.04% 101.16% 101.71% 91.86%
> sweep4 on 108.33% 101.69% 97.14% 100.92%
>
> The benefit varies with like +3-10% depending on connection count.
> Quite frankly I was expecting a little bit more, especially after
> re-reading [1]. Maybe you preloaded it there using pg_prewarm? (here
> I've randomly warmed it using pgbench). Probably it's something with
> my test, I'll take yet another look hopefully soon. The good thing is
> that it never crashed and I haven't seen any errors like "Bad address"
> probably related to AIO as you saw in [1], perhaps I wasn't using
> uring.
>
Hmmm. I'd have expected better results for this workload. So I tried
re-running my seqscan benchmark on the 176-core instance, and I got this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 43 35 40 53 53 46 45
96 55 54 42 47 57 58 53 53
128 59 59 46 50 58 58 57 60
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 98% 79% 92% 122% 122% 105% 104%
96 99% 76% 86% 104% 105% 97% 97%
128 99% 77% 84% 98% 98% 97% 101%
I did the benchmark for individual parts of the patch series. There's a
clear (~20%) speedup for 0005, but 0006 and 0007 make it go away. The
0002/0003 regress it quite a bit. And with 128 clients there's no
improvement at all.
This was with the default number of partitions (i.e. 4). If I increase
the number to 16, I get this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 43 69 82 87 87 78 79
96 55 54 65 85 91 91 86 86
128 59 59 66 77 83 83 82 86
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 99% 158% 189% 199% 199% 180% 180%
96 100% 119% 156% 167% 167% 157% 158%
128 99% 112% 130% 140% 140% 139% 145%
And with 32 partitions, I get this:
clients master 0001 0002 0003 0004 0005 0006 0007
-----------------------------------------------------------------
64 44 44 88 91 90 90 84 84
96 55 54 89 93 93 92 90 91
128 59 59 85 84 86 85 88 87
clients 0001 0002 0003 0004 0005 0006 0007
--------------------------------------------------------
64 100% 202% 208% 207% 207% 193% 193%
96 100% 163% 169% 171% 168% 165% 166%
128 99% 144% 142% 146% 144% 149% 146%
Those are clearly much better results, so I guess the default number of
partitions may be too low.
What bothers me is that this seems like a very narrow benchmark. I mean,
few systems are doing concurrent seqscans putting this much pressure on
buffer replacement. And once the plans start to do other stuff, the
contention on clock sweep seems to go down substantially (as shown by
the read-only pgbench). So the question is - is this really worth it?
> 0007 (PROCs) still complains with "mbind: Invalid argument" (aligment issue)
>
Should be fixed by the attached patches. The 0006 patch has an issue
with mbind too, but it was visible only when the buffers were not a nice
multiple of memory pages (and multiples of 1GB are fine).
This also moves the memset() until after placing the PGPROC partitions
to different NUMA nodes.
The results above are from v20251121. I'll rerun the tests with the nw
version of the patches. But it can only change the 0006/0007 results, of
course. The 0001-0005 are the same.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v20251126-0009-mbind-procs.patch (2.4K, ../[email protected]/2-v20251126-0009-mbind-procs.patch)
download | inline diff:
From 6a7b5f0c48b861a78baafad699802752edfce669 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 26 Nov 2025 17:13:09 +0100
Subject: [PATCH v20251126 9/9] mbind: procs
---
src/backend/storage/lmgr/proc.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 0a0ce98b725..3d76a3a6429 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -206,11 +206,14 @@ FastPathLockShmemSize(void)
* when not strictly needed (if it's already aligned). And we always
* assume we'll add a whole page, even if the alignment needs only less
* memory.
+ *
+ * XXX We need two extra pages. One for the non-NUMA part (aux processes),
+ * and one to keep the size of the last chunk aligned too.
*/
if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
Assert(numa_nodes > 0);
- size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ size = add_size(size, mul_size((numa_nodes + 2), numa_page_size));
}
return size;
@@ -324,6 +327,7 @@ InitProcGlobal(void)
/* Used for setup of per-backend fast-path slots. */
char *fpPtr,
+ *fpPtrOrig,
*fpEndPtr PG_USED_FOR_ASSERTS_ONLY;
Size fpLockBitsSize,
fpRelIdSize;
@@ -507,7 +511,7 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(fpPtr, 0, requestSize);
+ fpPtrOrig = fpPtr;
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
@@ -595,6 +599,9 @@ InitProcGlobal(void)
Assert(fpPtr <= fpEndPtr);
}
+ /* zero the memory only after locating the memory to NUMA nodes */
+ MemSet(fpPtrOrig, 0, requestSize);
+
for (i = 0; i < TotalProcs; i++)
{
PGPROC *proc = procs[i];
@@ -2521,7 +2528,10 @@ fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
* memory, to make sure it's not mapped to any node yet
*/
if (node != -1)
+ {
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
pg_numa_move_to_node(ptr, endptr, node);
+ }
/*
* Now point the PGPROC entries to the fast-path arrays, and also advance
@@ -2550,7 +2560,8 @@ fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
allprocs_index++;
}
- Assert(ptr == endptr);
+ Assert(ptr <= endptr);
+ Assert((node == -1) || (char *) TYPEALIGN(numa_page_size, ptr) == endptr);
return endptr;
}
--
2.51.1
[text/x-patch] v20251126-0008-NUMA-partition-PGPROC.patch (49.2K, ../[email protected]/3-v20251126-0008-NUMA-partition-PGPROC.patch)
download | inline diff:
From b6e4879a397210aff0f0f720708471568fe99ea4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:10:03 +0100
Subject: [PATCH v20251126 8/9] NUMA: partition PGPROC
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
Note: There's some challenges in making this work on EXEC_BACKEND, even
if we don't support NUMA on platforms that require this.
---
.../pg_buffercache--1.7--1.8.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/access/transam/twophase.c | 3 +-
src/backend/postmaster/launch_backend.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 +
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 85 ++-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 551 +++++++++++++++++-
src/include/port/pg_numa.h | 1 +
src/include/storage/proc.h | 18 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 72 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 43d2e84f9d2..265c35c8252 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -31,3 +31,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index eae75375152..c6629f767d1 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -30,6 +31,7 @@
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -105,6 +107,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -981,3 +984,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index ea43b432daf..7d589bac115 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -575,7 +575,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -634,7 +634,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 89d0bfa7760..e0e17293536 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -282,7 +282,7 @@ TwoPhaseShmemInit(void)
TwoPhaseState->freeGXacts = &gxacts[i];
/* associate it with a PGPROC assigned by InitProcGlobal */
- gxacts[i].pgprocno = GetNumberFromPGProc(&PreparedXactProcs[i]);
+ gxacts[i].pgprocno = GetNumberFromPGProc(PreparedXactProcs[i]);
}
}
else
@@ -447,6 +447,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, FullTransactionId fxid,
/* Initialize the PGPROC entry */
MemSet(proc, 0, sizeof(PGPROC));
+ proc->procnumber = gxact->pgprocno;
dlist_node_init(&proc->links);
proc->waitStatus = PROC_WAIT_STATUS_OK;
if (LocalTransactionIdIsValid(MyProc->vxid.lxid))
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 976638a58ac..5e7b0ac8850 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -107,8 +107,8 @@ typedef struct
LWLockPadded *MainLWLockArray;
slock_t *ProcStructLock;
PROC_HDR *ProcGlobal;
- PGPROC *AuxiliaryProcs;
- PGPROC *PreparedXactProcs;
+ PGPROC **AuxiliaryProcs;
+ PGPROC **PreparedXactProcs;
volatile PMSignalData *PMSignalState;
ProcSignalHeader *ProcSignal;
pid_t PostmasterPid;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..3288900bb6f 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -292,7 +292,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..f5844aa5b6a 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 9ba455488a0..55016cce93f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -824,6 +824,8 @@ check_debug_numa(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(item, "buffers") == 0)
flags |= NUMA_BUFFERS;
+ else if (pg_strcasecmp(item, "procs") == 0)
+ flags |= NUMA_PROCS;
else
{
GUC_check_errdetail("Invalid option \"%s\".", item);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 810a549efce..0937292643f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -472,7 +472,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..7e28fbdfea3 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -369,6 +369,8 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
TransactionId xid);
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
+static void AssertCheckAllProcs(void);
+
/*
* Report shared-memory space needed by ProcArrayShmemInit
*/
@@ -476,6 +478,8 @@ ProcArrayAdd(PGPROC *proc)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
if (arrayP->numProcs >= arrayP->maxProcs)
{
/*
@@ -502,7 +506,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,11 +542,13 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -578,10 +584,12 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
+ AssertCheckAllProcs();
+
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,11 +644,13 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
+ AssertCheckAllProcs();
+
/*
* Release in reversed acquisition order, to reduce frequency of having to
* wait for XidGenLock while holding ProcArrayLock.
@@ -860,7 +870,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +890,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1536,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1632,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1749,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2233,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2307,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2508,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2734,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2765,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2867,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3029,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3070,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3198,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3241,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3310,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3412,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3477,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3532,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3578,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3607,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3638,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3679,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3742,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3808,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
@@ -5227,3 +5236,15 @@ KnownAssignedXidsReset(void)
LWLockRelease(ProcArrayLock);
}
+
+static void
+AssertCheckAllProcs(void)
+{
+ ProcArrayStruct *arrayP = procArray;
+ int numProcs = arrayP->numProcs;
+
+ for (int pgxactoff = 0; pgxactoff < numProcs; pgxactoff++)
+ {
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
+ }
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 9cb78ead105..f82e664ad3f 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2876,7 +2876,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3135,7 +3135,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3822,7 +3822,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..0a0ce98b725 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,22 +29,33 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "access/xlogwait.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -77,8 +88,8 @@ NON_EXEC_STATIC slock_t *ProcStructLock = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
-NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
-PGPROC *PreparedXactProcs = NULL;
+NON_EXEC_STATIC PGPROC **AuxiliaryProcs = NULL;
+PGPROC **PreparedXactProcs = NULL;
static DeadLockState deadlock_state = DS_NOT_YET_CHECKED;
@@ -91,6 +102,29 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -101,11 +135,41 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, CACHELINEALIGN(mul_size(TotalProcs, sizeof(PGPROC *))));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -130,6 +194,60 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
+ return size;
+}
+
+static Size
+PGProcPartitionsShmemSize(void)
+{
+ Size size = 0;
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -141,6 +259,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -149,6 +270,8 @@ ProcGlobalShmemSize(void)
size = add_size(size, PGProcShmemSize());
size = add_size(size, FastPathLockShmemSize());
+ size = add_size(size, PGProcPartitionsShmemSize());
+
return size;
}
@@ -193,7 +316,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -212,6 +335,9 @@ InitProcGlobal(void)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
Assert(!found);
+ /* XXX call again, EXEC_BACKEND may not see the already computed value */
+ pgproc_partitions_prepare();
+
/*
* Initialize the data structures.
*/
@@ -226,6 +352,15 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
+ /* PGPROC partition registry */
+ requestSize = PGProcPartitionsShmemSize();
+
+ ptr = ShmemInitStruct("PGPROC partitions",
+ requestSize,
+ &found);
+
+ partitions = (PGProcPartition *) ptr;
+
/*
* Create and initialize all the PGPROC structures we'll need. There are
* six separate consumers: (1) normal backends, (2) autovacuum workers and
@@ -241,21 +376,110 @@ InitProcGlobal(void)
requestSize,
&found);
- MemSet(ptr, 0, requestSize);
-
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + CACHELINEALIGN(TotalProcs * sizeof(PGPROC *));
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partition
+ * per NUMA node (and then one extra partition for auxiliary procs).
+ */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ /* should have been aligned */
+ Assert(ptr == (char *) TYPEALIGN(numa_page_size, ptr));
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ /*
+ * Don't memset the memory before locating it to NUMA nodes (which requires
+ * the pages to be allocated but not yet faulted in memory).
+ */
+ MemSet(ptr, 0, requestSize);
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -291,23 +515,91 @@ InitProcGlobal(void)
/* Reserve space for semaphores. */
PGReserveSemaphores(ProcGlobalSemas());
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
- /* Common initialization for all PGPROCs, regardless of type. */
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
@@ -371,9 +663,6 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
@@ -440,7 +729,51 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if ((numa_flags & NUMA_PROCS) != 0)
+ {
+ dlist_mutable_iter iter;
+ int node;
+
+#ifdef USE_LIBNUMA
+ int cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ /* FIXME is defaulting to 0 correct? */
+ node = 0;
+#endif
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -651,7 +984,7 @@ InitAuxiliaryProcess(void)
*/
for (proctype = 0; proctype < NUM_AUXILIARY_PROCS; proctype++)
{
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
if (auxproc->pid == 0)
break;
}
@@ -1059,7 +1392,7 @@ AuxiliaryProcKill(int code, Datum arg)
if (MyProc->pid != (int) getpid())
elog(PANIC, "AuxiliaryProcKill() called in child process");
- auxproc = &AuxiliaryProcs[proctype];
+ auxproc = AuxiliaryProcs[proctype];
Assert(MyProc == auxproc);
@@ -1108,7 +1441,7 @@ AuxiliaryPidGetProc(int pid)
for (index = 0; index < NUM_AUXILIARY_PROCS; index++)
{
- PGPROC *proc = &AuxiliaryProcs[index];
+ PGPROC *proc = AuxiliaryProcs[index];
if (proc->pid == pid)
{
@@ -1998,7 +2331,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2073,3 +2406,173 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#ifdef USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ numa_nodes = 1;
+#endif
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ // Assert(!IsUnderPostmaster);
+
+ numa_page_size = pg_numa_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(DEBUG1, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+/*
+ *
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ {
+ /* align the pointer to the next page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+ }
+
+ elog(DEBUG1, "NUMA: pgproc_partition_init procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (((numa_flags & NUMA_PROCS) != 0) && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9734aa315ff..aa524f6f7f3 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -23,6 +23,7 @@ extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int n
extern PGDLLIMPORT int numa_flags;
#define NUMA_BUFFERS 0x01
+#define NUMA_PROCS 0x02
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..21f2619fd40 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -438,13 +443,13 @@ typedef struct PROC_HDR
extern PGDLLIMPORT PROC_HDR *ProcGlobal;
-extern PGDLLIMPORT PGPROC *PreparedXactProcs;
+extern PGDLLIMPORT PGPROC **PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -480,7 +485,7 @@ extern PGDLLIMPORT bool log_lock_waits;
#ifdef EXEC_BACKEND
extern PGDLLIMPORT slock_t *ProcStructLock;
-extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
+extern PGDLLIMPORT PGPROC **AuxiliaryProcs;
#endif
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241f175e9da..e1bf02a3567 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1888,6 +1888,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.51.1
[text/x-patch] v20251126-0007-mbind-buffers.patch (2.8K, ../[email protected]/4-v20251126-0007-mbind-buffers.patch)
download | inline diff:
From 191b5b6503c0884260d14479fe5b19fc1cc7cc86 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 26 Nov 2025 17:12:57 +0100
Subject: [PATCH v20251126 7/9] mbind: buffers
---
src/backend/storage/buffer/buf_init.c | 29 +++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 587859a5754..9ba455488a0 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -272,6 +272,13 @@ BufferManagerShmemSize(void)
size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
+ /*
+ * Extra alignment, so that the partitions are whole memory pages (we
+ * may need to pad the last one, so one page is enough). Without this
+ * we may get mbind() failures in pg_numa_move_to_node().
+ */
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
+
/* size of stuff controlled by freelist.c */
size = add_size(size, StrategyShmemSize());
@@ -695,6 +702,15 @@ buffer_partitions_init(void)
/* first map buffers */
startptr = buffers_ptr;
endptr = startptr + ((Size) num_buffers * BLCKSZ);
+
+ /*
+ * Make sure the partition is a multiple of memory page, so that we
+ * don't get mbind failures in move_to_node calls. This matters only
+ * for the last partition, the earlier ones should be always sized
+ * as multiples of pages.
+ */
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+
buffers_ptr = endptr; /* start of the next partition */
elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
@@ -705,6 +721,15 @@ buffer_partitions_init(void)
/* now do the same for buffer descriptors */
startptr = descriptors_ptr;
endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+
+ /*
+ * Make sure the partition is a multiple of memory page, so that we
+ * don't get mbind failures in move_to_node calls. This matters only
+ * for the last partition, the earlier ones should be always sized
+ * as multiples of pages.
+ */
+ endptr = (char *) TYPEALIGN(numa_page_size, endptr);
+
descriptors_ptr = endptr;
elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
@@ -714,8 +739,8 @@ buffer_partitions_init(void)
}
/* we should have consumed the arrays exactly */
- Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
- Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+ Assert(buffers_ptr <= (char *) TYPEALIGN(numa_page_size, BufferBlocks + (Size) NBuffers * BLCKSZ));
+ Assert(descriptors_ptr == (char *) TYPEALIGN(numa_page_size, (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded)));
}
int
--
2.51.1
[text/x-patch] v20251126-0006-NUMA-shared-buffers-partitioning.patch (47.0K, ../[email protected]/5-v20251126-0006-NUMA-shared-buffers-partitioning.patch)
download | inline diff:
From 4b3779f92217900c0eea91ef68810867362a7170 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:05:35 +0100
Subject: [PATCH v20251126 6/9] NUMA: shared buffers partitioning
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
Notes:
* The feature is enabled by debug_numa = buffers GUC (default: empty),
which works similarly to debug_io_direct.
* This patch partitions just shared buffers, not the whole shared
memory. A later patch will do that for PGPROC, but it's tricky and
requires a different approach because of huge pages.
---
.../pg_buffercache--1.7--1.8.sql | 1 +
contrib/pg_buffercache/pg_buffercache_pages.c | 52 +-
src/backend/port/sysv_shmem.c | 34 +-
src/backend/storage/buffer/buf_init.c | 569 +++++++++++++++++-
src/backend/storage/buffer/freelist.c | 88 ++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/guc_tables.c | 1 +
src/include/port/pg_numa.h | 6 +
src/include/storage/buf_internals.h | 14 +-
src/include/storage/bufmgr.h | 4 +
src/include/utils/guc_hooks.h | 3 +
src/port/pg_numa.c | 64 ++
12 files changed, 773 insertions(+), 73 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index 1834599c4b3..43d2e84f9d2 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -11,6 +11,7 @@ LANGUAGE C PARALLEL SAFE;
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
(partition integer, -- partition index
+ numa_node integer, -- NUMA node of the partitioon
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
last_buffer integer, -- last buffer of partition
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 81665209084..eae75375152 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -29,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 12
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -863,25 +863,27 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "num_passes",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "next_buffer",
INT4OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "total_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "num_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "total_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "num_req_allocs",
INT8OID, -1, 0);
- TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ TupleDescInitEntry(tupledesc, (AttrNumber) 12, "weigths",
typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -899,7 +901,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
{
uint32 i = funcctx->call_cntr;
- int num_buffers,
+ int numa_node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -918,7 +921,7 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
- BufferPartitionGet(i, &num_buffers,
+ BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
@@ -936,36 +939,39 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[0] = Int32GetDatum(i);
nulls[0] = false;
- values[1] = Int32GetDatum(num_buffers);
+ values[1] = Int32GetDatum(numa_node);
nulls[1] = false;
- values[2] = Int32GetDatum(first_buffer);
+ values[2] = Int32GetDatum(num_buffers);
nulls[2] = false;
- values[3] = Int32GetDatum(last_buffer);
+ values[3] = Int32GetDatum(first_buffer);
nulls[3] = false;
- values[4] = Int64GetDatum(complete_passes);
+ values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
- values[5] = Int32GetDatum(next_victim_buffer);
+ values[5] = Int64GetDatum(complete_passes);
nulls[5] = false;
- values[6] = Int64GetDatum(buffer_total_allocs);
+ values[6] = Int32GetDatum(next_victim_buffer);
nulls[6] = false;
- values[7] = Int64GetDatum(buffer_allocs);
+ values[7] = Int64GetDatum(buffer_total_allocs);
nulls[7] = false;
- values[8] = Int64GetDatum(buffer_total_req_allocs);
+ values[8] = Int64GetDatum(buffer_allocs);
nulls[8] = false;
- values[9] = Int64GetDatum(buffer_req_allocs);
+ values[9] = Int64GetDatum(buffer_total_req_allocs);
nulls[9] = false;
- values[10] = PointerGetDatum(array);
+ values[10] = Int64GetDatum(buffer_req_allocs);
nulls[10] = false;
+ values[11] = PointerGetDatum(array);
+ nulls[11] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..78a0c5199f1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include <numa.h>
#include <signal.h>
#include <unistd.h>
#include <sys/file.h>
@@ -602,6 +603,14 @@ CreateAnonymousSegment(Size *size)
void *ptr = MAP_FAILED;
int mmap_errno = 0;
+ /*
+ * Set the memory policy to interleave to all NUMA nodes before calling
+ * mmap, in case we use MAP_POPULATE to prefault all the pages.
+ *
+ * XXX Probably not needed without that, but also costs nothing.
+ */
+ numa_set_membind(numa_all_nodes_ptr);
+
#ifndef MAP_HUGETLB
/* PGSharedMemoryCreate should have dealt with this case */
Assert(huge_pages != HUGE_PAGES_ON);
@@ -616,6 +625,9 @@ CreateAnonymousSegment(Size *size)
GetHugePageSize(&hugepagesize, &mmap_flags);
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
if (allocsize % hugepagesize != 0)
allocsize += hugepagesize - (allocsize % hugepagesize);
@@ -638,13 +650,18 @@ CreateAnonymousSegment(Size *size)
if (ptr == MAP_FAILED && huge_pages != HUGE_PAGES_ON)
{
+ int mmap_flags = 0;
+
+ // prefault the memory at start?
+ // mmap_flags |= MAP_POPULATE;
+
/*
* Use the original size, not the rounded-up value, when falling back
* to non-huge pages.
*/
allocsize = *size;
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS, -1, 0);
+ PG_MMAP_FLAGS | mmap_flags, -1, 0);
mmap_errno = errno;
}
@@ -663,6 +680,21 @@ CreateAnonymousSegment(Size *size)
allocsize) : 0));
}
+ /* undo the earlier num_set_membind() call. */
+ numa_set_localalloc();
+
+ /*
+ * Before touching the memory, set the allocation policy, so that
+ * it gets interleaved by default. We have to do this to distribute
+ * the memory that's not located explicitly. We need this especially
+ * with huge pages, where we could run out of huge pages on some
+ * nodes and crash otherwise.
+ *
+ * XXX Probably not needed with MAP_POPULATE, in which case the policy
+ * was already set by num_set_membind() earlier. But doesn't hurt.
+ */
+ numa_interleave_memory(ptr, allocsize, numa_all_nodes_ptr);
+
*size = allocsize;
return ptr;
}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 0362fda24aa..587859a5754 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,12 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -29,15 +35,24 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
-/* *
- * number of buffer partitions */
-#define NUM_CLOCK_SWEEP_PARTITIONS 4
+/*
+ * Minimum number of buffer partitions, no matter the number of NUMA nodes.
+ */
+#define MIN_BUFFER_PARTITIONS 4
/* Array of structs with information about buffer ranges */
BufferPartitions *BufferPartitionsArray = NULL;
+static void buffer_partitions_prepare(void);
static void buffer_partitions_init(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
* buffers live in a freelist and a lookup data structure.
@@ -85,25 +100,85 @@ BufferManagerShmemInit(void)
foundIOCV,
foundBufCkpt,
foundParts;
+ Size buffer_align;
+
+ /*
+ * Determine the memory page size used to partition shared buffers over
+ * the available NUMA nodes.
+ *
+ * XXX We have to call prepare again, because with EXEC_BACKEND we may not
+ * see the values already calculated in BufferManagerShmemSize().
+ *
+ * XXX We need to be careful to get the same value when calculating the
+ * and then later when initializing the structs after allocation, or to not
+ * depend on that value too much. Before the allocation we don't know if we
+ * get huge pages, so we just have to assume we do.
+ */
+ buffer_partitions_prepare();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size. NUMA works on page
+ * granularity, and we don't want a buffer to get split to multiple nodes
+ * (when spanning multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (numa_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to numa_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(numa_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((numa_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % numa_page_size) == 0));
/* allocate the partition registry first */
BufferPartitionsArray = (BufferPartitions *)
ShmemInitStruct("Buffer Partitions",
offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ mul_size(sizeof(BufferPartition), numa_partitions),
&foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -133,7 +208,10 @@ BufferManagerShmemInit(void)
{
int i;
- /* Initialize buffer partitions (calculate buffer ranges). */
+ /*
+ * Initialize buffer partitions, including moving memory to different
+ * NUMA nodes (if enabled by GUC).
+ */
buffer_partitions_init();
/*
@@ -172,19 +250,26 @@ BufferManagerShmemInit(void)
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -201,11 +286,244 @@ BufferManagerShmemSize(void)
/* account for registry of NUMA partitions */
size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
- mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+ mul_size(sizeof(BufferPartition), numa_partitions)));
return size;
}
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA partitioning */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ /* no NUMA-aware partitioning */
+ if ((numa_flags & NUMA_BUFFERS) == 0)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches earlier in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+#if USE_LIBNUMA
+ numa_nodes = numa_num_configured_nodes();
+#else
+ /* without NUMA, assume there's just one node */
+ numa_nodes = 1;
+#endif
+
+ /* we should never get here without at least one NUMA node */
+ Assert(numa_nodes > 0);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ numa_page_size = pg_numa_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(NOTICE, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+ else if ((numa_flags & NUMA_BUFFERS) == 0)
+ {
+ elog(NOTICE, "NUMA-partitioning of buffers disabled");
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or when we can't partition for some
+ * reason, just take a "fair share" of buffers. This can happen for a
+ * number of reasons - missing NUMA support, partitioning of buffers not
+ * enabled, or not enough buffers for this many nodes.
+ *
+ * We still build partitions, because we want to allow partitioning of
+ * the clock-sweep later.
+ *
+ * The number of buffers for each partition is calculated later, once we
+ * have allocated the shared memory (because that's where we store it).
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(DEBUG1, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
/*
* Sanity checks of buffers partitions - there must be no gaps, it must cover
* the whole range of buffers, etc.
@@ -267,33 +585,137 @@ buffer_partitions_init(void)
{
int remaining_buffers = NBuffers;
int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
- /* number of buffers per partition (make sure to not overflow) */
- int part_buffers
- = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
-
- BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ BufferPartitionsArray->npartitions = numa_partitions;
+ BufferPartitionsArray->nnodes = numa_nodes;
- for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ for (int n = 0; n < numa_nodes; n++)
{
- BufferPartition *part = &BufferPartitionsArray->partitions[n];
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
- /* buffers this partition should get (last partition can get fewer) */
- int num_buffers = Min(remaining_buffers, part_buffers);
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
- remaining_buffers -= num_buffers;
+ remaining_buffers -= node_buffers;
- Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
- part->num_buffers = num_buffers;
- part->first_buffer = buffer;
- part->last_buffer = buffer + (num_buffers - 1);
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
- buffer += num_buffers;
+ /* XXX we should get the actual node ID from the mask */
+ if (numa_can_partition)
+ part->numa_node = n;
+ else
+ part->numa_node = -1;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(DEBUG1, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
}
AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (((numa_flags & NUMA_BUFFERS) == 0) || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not map pages
+ * one by one.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ *
+ * We always map all partitions for the same node at once, so that we
+ * don't need to worry about alignment of memory pages that get split
+ * between partitions (we only worry about min_node_buffers for whole
+ * NUMA nodes, not for individual partitions).
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ char *startptr,
+ *endptr;
+ int num_buffers = 0;
+
+ /* sum buffers in all partitions for this node */
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int pidx = (n * parts_per_node + p);
+ BufferPartition *part = &BufferPartitionsArray->partitions[pidx];
+
+ Assert(part->numa_node == n);
+
+ num_buffers += part->num_buffers;
+ }
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => buffers %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(DEBUG1, "NUMA: buffer_partitions_init: %d => descriptors %d start %p end %p (size %zd)",
+ n, num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, n);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
}
int
@@ -302,14 +724,21 @@ BufferPartitionCount(void)
return BufferPartitionsArray->npartitions;
}
+int
+BufferPartitionNodes(void)
+{
+ return BufferPartitionsArray->nnodes;
+}
+
void
-BufferPartitionGet(int idx, int *num_buffers,
+BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer)
{
if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
{
BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ *node = part->numa_node;
*num_buffers = part->num_buffers;
*first_buffer = part->first_buffer;
*last_buffer = part->last_buffer;
@@ -322,8 +751,82 @@ BufferPartitionGet(int idx, int *num_buffers,
/* return parameters before the partitions are initialized (during sizing) */
void
-BufferPartitionParams(int *num_partitions)
+BufferPartitionParams(int *num_partitions, int *num_nodes)
{
if (num_partitions)
- *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+ *num_partitions = numa_partitions;
+
+ if (num_nodes)
+ *num_nodes = numa_nodes;
+}
+
+/* XXX the GUC hooks should probably be somewhere else? */
+bool
+check_debug_numa(char **newval, void **extra, GucSource source)
+{
+ bool result = true;
+ int flags;
+
+#if USE_LIBNUMA == 0
+ if (strcmp(*newval, "") != 0)
+ {
+ GUC_check_errdetail("\"%s\" is not supported on this platform.",
+ "debug_numa");
+ result = false;
+ }
+ flags = 0;
+#else
+ List *elemlist;
+ ListCell *l;
+ char *rawstring;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "debug_numa");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ flags = 0;
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "buffers") == 0)
+ flags |= NUMA_BUFFERS;
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ result = false;
+ break;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+#endif
+
+ if (!result)
+ return result;
+
+ /* Save the flags in *extra, for use by assign_debug_io_direct */
+ *extra = guc_malloc(LOG, sizeof(int));
+ if (!*extra)
+ return false;
+ *((int *) *extra) = flags;
+
+ return result;
+}
+
+void
+assign_debug_numa(const char *newval, void *extra)
+{
+ int *flags = (int *) extra;
+
+ numa_flags = *flags;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 8be77a9c8b1..810a549efce 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -124,7 +124,9 @@ typedef struct
//int __attribute__((aligned(64))) bgwprocno;
/* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
int num_partitions;
+ int num_partitions_per_node;
/* clocksweep partitions */
ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
@@ -270,16 +272,72 @@ ClockSweepTick(ClockSweep *sweep)
* calculate_partition_index
* calculate the buffer / clock-sweep partition to use
*
- * use PID to determine the buffer partition
- *
- * XXX We could use NUMA node / core ID to pick partition, but we'd need
- * to handle cases with fewer nodes/cores than partitions somehow. Although,
- * maybe the balancing would handle that too.
+ * With libnuma, use the NUMA node and CPU to pick the partition. Otherwise
+ * use just PID instead of CPU (we assume everything is a single NUMA node).
*/
static int
calculate_partition_index(void)
{
- return (MyProcPid % StrategyControl->num_partitions);
+ int cpu,
+ node,
+ index;
+
+ /*
+ * The buffers are partitioned, so determine the CPU/NUMA node, and pick a
+ * partition based on that.
+ *
+ * Without NUMA assume everything is a single NUMA node, and we pick the
+ * partition based on PID (we may not have sched_getcpu).
+ */
+#ifdef USE_LIBNUMA
+ cpu = sched_getcpu();
+
+ if (cpu < 0)
+ elog(ERROR, "sched_getcpu failed: %m");
+
+ node = numa_node_of_cpu(cpu);
+#else
+ cpu = MyProcPid;
+ node = 0;
+#endif
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ /* fast-path */
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
}
/*
@@ -947,7 +1005,7 @@ StrategyShmemSize(void)
Size size = 0;
int num_partitions;
- BufferPartitionParams(&num_partitions);
+ BufferPartitionParams(&num_partitions, NULL);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -974,9 +1032,17 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
int num_partitions;
+ int num_partitions_per_node;
num_partitions = BufferPartitionCount();
+ num_nodes = BufferPartitionNodes();
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
/*
* Initialize the shared buffer lookup hashtable.
@@ -1011,7 +1077,8 @@ StrategyInitialize(bool init)
/* Initialize the clock sweep pointers (for all partitions) */
for (int i = 0; i < num_partitions; i++)
{
- int num_buffers,
+ int node,
+ num_buffers,
first_buffer,
last_buffer;
@@ -1020,7 +1087,8 @@ StrategyInitialize(bool init)
pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
/* get info about the buffer partition */
- BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
/*
* FIXME This may not quite right, because if NBuffers is not a
@@ -1056,6 +1124,8 @@ StrategyInitialize(bool init)
/* initialize the partitioned clocksweep */
StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
}
else
Assert(!init);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 3b9d8349078..b13e667ac9e 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -643,6 +643,16 @@
options => 'debug_logical_replication_streaming_options',
},
+{ name => 'debug_numa', type => 'string', context => 'PGC_POSTMASTER', group => 'DEVELOPER_OPTIONS',
+ short_desc => 'NUMA-aware partitioning of shared memory.',
+ long_desc => 'An empty string disables NUMA-aware partitioning.',
+ flags => 'GUC_LIST_INPUT | GUC_NOT_IN_SAMPLE',
+ variable => 'debug_numa_string',
+ boot_val => '""',
+ check_hook => 'check_debug_numa',
+ assign_hook => 'assign_debug_numa',
+},
+
{ name => 'debug_parallel_query', type => 'enum', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS',
short_desc => 'Forces the planner\'s use parallel query nodes.',
long_desc => 'This can be useful for testing the parallel query infrastructure by forcing the planner to generate plans that contain nodes that perform tuple communication between workers and the main process.',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f87b558c2c6..3f29eeaf5ae 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -595,6 +595,7 @@ static char *server_version_string;
static int server_version_num;
static char *debug_io_direct_string;
static char *restrict_nonsystem_relation_kind_string;
+static char *debug_numa_string;
#ifdef HAVE_SYSLOG
#define DEFAULT_SYSLOG_FACILITY LOG_LOCAL0
diff --git a/src/include/port/pg_numa.h b/src/include/port/pg_numa.h
index 9d1ea6d0db8..9734aa315ff 100644
--- a/src/include/port/pg_numa.h
+++ b/src/include/port/pg_numa.h
@@ -17,6 +17,12 @@
extern PGDLLIMPORT int pg_numa_init(void);
extern PGDLLIMPORT int pg_numa_query_pages(int pid, unsigned long count, void **pages, int *status);
extern PGDLLIMPORT int pg_numa_get_max_node(void);
+extern PGDLLIMPORT Size pg_numa_page_size(void);
+extern PGDLLIMPORT void pg_numa_move_to_node(char *startptr, char *endptr, int node);
+
+extern PGDLLIMPORT int numa_flags;
+
+#define NUMA_BUFFERS 0x01
#ifdef USE_LIBNUMA
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1118b386228..33377841c57 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -299,10 +299,10 @@ typedef struct BufferDesc
* line sized.
*
* XXX: As this is primarily matters in highly concurrent workloads which
- * probably all are 64bit these days, and the space wastage would be a bit
- * more noticeable on 32bit systems, we don't force the stride to be cache
- * line sized on those. If somebody does actual performance testing, we can
- * reevaluate.
+ * probably all are 64bit these days. We force the stride to be cache line
+ * sized even on 32bit systems, where the space wastage is be a bit more
+ * noticeable, to allow partitioning of shared buffers (which requires the
+ * memory page be a multiple of buffer descriptor).
*
* Note that local buffer descriptors aren't forced to be aligned - as there's
* no concurrent access to those it's unlikely to be beneficial.
@@ -312,7 +312,7 @@ typedef struct BufferDesc
* platform with either 32 or 128 byte line sizes, it's good to align to
* boundaries and avoid false sharing.
*/
-#define BUFFERDESC_PAD_TO_SIZE (SIZEOF_VOID_P == 8 ? 64 : 1)
+#define BUFFERDESC_PAD_TO_SIZE 64
typedef union BufferDescPadded
{
@@ -555,8 +555,8 @@ extern void AtEOXact_LocalBuffers(bool isCommit);
extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
-extern void BufferPartitionGet(int idx, int *num_buffers,
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
int *first_buffer, int *last_buffer);
-extern void BufferPartitionParams(int *num_partitions);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4e7b1fcd4ab..510018db115 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,10 +156,12 @@ typedef struct ReadBuffersOperation ReadBuffersOperation;
/*
* information about one partition of shared buffers
*
+ * numa_nod specifies node for this partition (-1 means allocated on any node)
* first/last buffer - the values are inclusive
*/
typedef struct BufferPartition
{
+ int numa_node; /* NUMA node (-1 no node) */
int num_buffers; /* number of buffers */
int first_buffer; /* first buffer of partition */
int last_buffer; /* last buffer of partition */
@@ -169,6 +171,7 @@ typedef struct BufferPartition
typedef struct BufferPartitions
{
int npartitions; /* number of partitions */
+ int nnodes; /* number of NUMA nodes */
BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
} BufferPartitions;
@@ -346,6 +349,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 82ac8646a8d..15304df0de5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -175,4 +175,7 @@ extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_debug_numa(char **newval, void **extra, GucSource source);
+extern void assign_debug_numa(const char *newval, void *extra);
+
#endif /* GUC_HOOKS_H */
diff --git a/src/port/pg_numa.c b/src/port/pg_numa.c
index 540ada3f8ef..d9c3841e078 100644
--- a/src/port/pg_numa.c
+++ b/src/port/pg_numa.c
@@ -18,6 +18,9 @@
#include "miscadmin.h"
#include "port/pg_numa.h"
+#include "storage/pg_shmem.h"
+
+int numa_flags;
/*
* At this point we provide support only for Linux thanks to libnuma, but in
@@ -116,6 +119,36 @@ pg_numa_get_max_node(void)
return numa_max_node();
}
+/*
+ * pg_numa_move_to_node
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * node - node to move the memory to
+ *
+ * The "startptr" is expected to be a multiple of system memory page size, as
+ * determined by pg_numa_page_size.
+ *
+ * XXX We only expect to do this during startup, when the shared memory is
+ * still being setup.
+ */
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size sz = (endptr - startptr);
+
+ Assert((int64) startptr % pg_numa_page_size() == 0);
+
+ /*
+ * numa_tonode_memory does not actually cause a page fault, and thus does
+ * not locate the memory on the node. So it's fast, at least compared to
+ * pg_numa_query_pages, and does not make startup longer. But it also
+ * means the expensive part happen later, on the first access.
+ */
+ numa_tonode_memory(startptr, sz, node);
+}
+
#else
/* Empty wrappers */
@@ -138,4 +171,35 @@ pg_numa_get_max_node(void)
return 0;
}
+void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ /* we don't expect to ever get here in builds without libnuma */
+ Assert(false);
+}
+
#endif
+
+Size
+pg_numa_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
--
2.51.1
[text/x-patch] v20251126-0005-clock-sweep-weighted-balancing.patch (5.2K, ../[email protected]/6-v20251126-0005-clock-sweep-weighted-balancing.patch)
download | inline diff:
From 16013b08d9a524493c455278b1f4d97f237ec99e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 6 Aug 2025 01:09:57 +0200
Subject: [PATCH v20251126 5/9] clock-sweep: weighted balancing
The partitions may not be of exactly the same size, so consider that
when balancing clocksweep allocations.
Note: This may be more important with NUMA-aware partitioning, which
restricts the allowed sizes of partiions (especially with huge pages).
---
src/backend/storage/buffer/freelist.c | 63 ++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 3af82e267c6..8be77a9c8b1 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -619,6 +619,20 @@ StrategySyncBalance(void)
avg_allocs, /* average allocations (per partition) */
delta_allocs = 0; /* sum of allocs above average */
+ /*
+ * Size of a partition, used to calculate weighted average (the first
+ * partition is expected to be the largest one, and so will be counted
+ * as a "unit" partition with weight 1.0).
+ */
+ int32 num_buffers = StrategyControl->sweeps[0].numBuffers;
+
+ /*
+ * Total weight of partitions. If the partitions have the same size,
+ * the weight should be equal the partition count (modulo rounding
+ * errors, etc.)
+ */
+ double weight = 0.0;
+
/*
* Collect the number of allocations requested in the past interval.
* While at it, reset the counter to start the new interval.
@@ -645,16 +659,27 @@ StrategySyncBalance(void)
pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
total_allocs += allocs[i];
+
+ /* weight of the partition, relative to the "unit" partition */
+ weight += (sweep->numBuffers * 1.0 / num_buffers);
}
/*
- * Calculate the "fair share" of allocations per partition.
+ * XXX Not sure if the total_weight might exceed num_partitions due to
+ * rounding errors.
+ */
+ Assert((weight > 0.0) && (weight <= StrategyControl->num_partitions));
+
+ /*
+ * Calculate the "fair share" of allocations per partition. This is the
+ * number of allocations for the "unit" partition with num_buffers, we'll
+ * need to adjust it using the per-partition weight.
*
* XXX The last partition could be smaller, in which case it should be
* expected to handle fewer allocations. So this should be a weighted
* average. But for now a simple average is good enough.
*/
- avg_allocs = (total_allocs / StrategyControl->num_partitions);
+ avg_allocs = (total_allocs / weight);
/*
* Calculate the "delta" from balanced state, i.e. how many allocations
@@ -662,8 +687,14 @@ StrategySyncBalance(void)
*/
for (int i = 0; i < StrategyControl->num_partitions; i++)
{
- if (allocs[i] > avg_allocs)
- delta_allocs += (allocs[i] - avg_allocs);
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
+ if (allocs[i] > part_allocs)
+ delta_allocs += (allocs[i] - part_allocs);
}
/*
@@ -726,6 +757,10 @@ StrategySyncBalance(void)
ClockSweep *sweep = &StrategyControl->sweeps[i];
uint8 balance[MAX_BUFFER_PARTITIONS];
+ /* number of allocations expected for this partition */
+ double part_weight = (sweep->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs = avg_allocs * part_weight;
+
/* lock, we're going to modify the balance weights */
SpinLockAcquire(&sweep->clock_sweep_lock);
@@ -733,7 +768,7 @@ StrategySyncBalance(void)
memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
/* does this partition has fewer or more than avg_allocs? */
- if (allocs[i] < avg_allocs)
+ if (allocs[i] < part_allocs)
{
/* fewer - don't redirect any allocations elsewhere */
balance[i] = 100;
@@ -747,22 +782,30 @@ StrategySyncBalance(void)
* a fraction proportional to (excess/delta) from this one.
*/
- /* fraction of the "total" delta */
- double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+ /* fraction of the "total" delta represented by "excess" allocations */
+ double delta_frac = (allocs[i] - part_allocs) * 1.0 / delta_allocs;
/* keep just enough allocations to meet the target */
- balance[i] = (100.0 * avg_allocs / allocs[i]);
+ balance[i] = (100.0 * part_allocs / allocs[i]);
/* redirect the extra allocations */
for (int j = 0; j < StrategyControl->num_partitions; j++)
{
+ ClockSweep *sweep2 = &StrategyControl->sweeps[j];
+
+ /* number of allocations expected for this partition */
+ double part_weight_2 = (sweep2->numBuffers * 1.0 / num_buffers);
+ uint32 part_allocs_2 = avg_allocs * part_weight_2;
+
/* How many allocations to receive from i-th partition? */
- uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+ uint32 receive_allocs = delta_frac * (part_allocs_2 - allocs[j]);
/* ignore partitions that don't need additional allocations */
- if (allocs[j] > avg_allocs)
+ if (allocs[j] > part_allocs_2)
continue;
+ Assert(receive_allocs >= 0);
+
/* fraction to redirect */
balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
}
--
2.51.1
[text/x-patch] v20251126-0004-clock-sweep-scan-all-partitions.patch (6.7K, ../[email protected]/7-v20251126-0004-clock-sweep-scan-all-partitions.patch)
download | inline diff:
From 5df379b3a4f9773de589120c3a609d2008663620 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 15 Oct 2025 13:59:29 +0200
Subject: [PATCH v20251126 4/9] clock-sweep: scan all partitions
When looking for a free buffer, scan all clock-sweep partitions, not
just the "home" one. All buffers in the home partition may be pinned, in
which case we should not fail. Instead, advance to the next partition,
in a round-robin way, and only fail after scanning through all of them.
---
src/backend/storage/buffer/freelist.c | 91 ++++++++++++++++-------
src/test/recovery/t/027_stream_regress.pl | 5 --
2 files changed, 63 insertions(+), 33 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 169071032b4..3af82e267c6 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -167,6 +167,9 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
static ClockSweep *ChooseClockSweep(bool balance);
+static BufferDesc *StrategyGetBufferPartition(ClockSweep *sweep,
+ BufferAccessStrategy strategy,
+ uint32 *buf_state);
/*
* clocksweep allocation balancing
@@ -201,10 +204,9 @@ static int clocksweep_count = 0;
* id of the buffer now under the hand.
*/
static inline uint32
-ClockSweepTick(void)
+ClockSweepTick(ClockSweep *sweep)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -370,7 +372,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
+ ClockSweep *sweep,
+ *sweep_start; /* starting clock-sweep partition */
*from_ring = false;
@@ -424,37 +427,69 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Use the "clock sweep" algorithm to find a free buffer
*
- * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
- * buffers from a single partition, aligned with the NUMA node. That means
- * it only accesses buffers from the same NUMA node.
- *
- * XXX That also means each process "sweeps" only a fraction of buffers,
- * even if the other buffers are better candidates for eviction. Maybe
- * there should be some logic to "steal" buffers from other freelists or
- * other nodes?
+ * Start with the "preferred" partition, and then proceed in a round-robin
+ * manner. If we cycle back to the starting partition, it means none of the
+ * partitions has unpinned buffers.
*
- * XXX Would that also mean we'd have multiple bgwriters, one for each
- * node, or would one bgwriter handle all of that?
+ * XXX Does this need to do similar balancing "balancing" as for bgwriter
+ * in StrategySyncBalance? Maybe it's be enough to simply pick the initial
+ * partition that way? We'd only getting a single buffer, so not much chance
+ * to balance over many allocations.
*
- * XXX This only searches a single partition, which can result in "no
- * unpinned buffers available" even if there are buffers in other
- * partitions. Should be fixed by falling back to other partitions if
- * needed.
- *
- * XXX Also, the trycounter should not be set to NBuffers, but to buffer
- * count for that one partition. In fact, this should not call ClockSweepTick
- * for every iteration. The call is likely quite expensive (does a lot
- * of stuff), and also may return a different partition on each call.
- * We should just do it once, and then do the for(;;) loop. And then
- * maybe advance to the next partition, until we scan through all of them.
+ * XXX But actually, we're calling ChooseClockSweep() with balance=true, so
+ * maybe it already does balancing?
*/
- trycounter = NBuffers;
+ sweep = ChooseClockSweep(true);
+ sweep_start = sweep;
+ for (;;)
+ {
+ buf = StrategyGetBufferPartition(sweep, strategy, buf_state);
+
+ /* found a buffer in the "sweep" partition, we're done */
+ if (buf != NULL)
+ return buf;
+
+ /*
+ * Try advancing to the next partition, round-robin (if last partition,
+ * wrap around to the beginning).
+ *
+ * XXX This is a bit ugly, there must be a better way to advance to the
+ * next partition.
+ */
+ if (sweep == &StrategyControl->sweeps[StrategyControl->num_partitions - 1])
+ sweep = StrategyControl->sweeps;
+ else
+ sweep++;
+
+ /* we've scanned all partitions */
+ if (sweep == sweep_start)
+ break;
+ }
+
+ /* we shouldn't get here if there are unpinned buffers */
+ elog(ERROR, "no unpinned buffers available");
+}
+
+/*
+ * StrategyGetBufferPartition
+ * get a free buffer from a single clock-sweep partition
+ *
+ * Returns NULL if there are no free (unpinned) buffers in the partition.
+*/
+static BufferDesc *
+StrategyGetBufferPartition(ClockSweep *sweep, BufferAccessStrategy strategy,
+ uint32 *buf_state)
+{
+ BufferDesc *buf;
+ int trycounter;
+
+ trycounter = sweep->numBuffers;
for (;;)
{
uint32 old_buf_state;
uint32 local_buf_state;
- buf = GetBufferDescriptor(ClockSweepTick());
+ buf = GetBufferDescriptor(ClockSweepTick(sweep));
/*
* Check whether the buffer can be used and pin it if so. Do this
@@ -482,7 +517,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* one eventually, but it's probably better to fail than
* to risk getting stuck in an infinite loop.
*/
- elog(ERROR, "no unpinned buffers available");
+ return NULL;
}
break;
}
@@ -501,7 +536,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state,
local_buf_state))
{
- trycounter = NBuffers;
+ trycounter = sweep->numBuffers;
break;
}
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 98b146ed4b7..589c79d97d3 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,11 +18,6 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
-# The default is 1MB, which is not enough with clock-sweep partitioning.
-# Increase to 32MB, so that we don't get "no unpinned buffers".
-$node_primary->append_conf('postgresql.conf',
- 'shared_buffers = 32MB');
-
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
--
2.51.1
[text/x-patch] v20251126-0003-clock-sweep-balancing-of-allocations.patch (25.3K, ../[email protected]/8-v20251126-0003-clock-sweep-balancing-of-allocations.patch)
download | inline diff:
From d5e333e0300db911e499e9bf2a7db95ff46e674d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 29 Oct 2025 21:45:34 +0100
Subject: [PATCH v20251126 3/9] clock-sweep: balancing of allocations
If backends only allocate buffers from the "home" partition, that may
cause significant misbalance. Some partitions might be overused, while
other partitions would be left unused. In other words, shared buffers
would not be used efficiently.
We want all partitions to be used about the same, i.e. serve about the
same number of allocations. To achieve that, allocations from partitions
that are "too busy" may get redirected to other partitions. The system
counts allocations requested from each partition, calculates the "fair
share" (average per partition), and then redirectsexcess allocations to
other partitions.
Each partition gets a set of coefficients determining the fraction of
allocations to redirect to other partitions. The coefficients may be
interpreted as a "budget" for each of the partition, i.e. the number of
allocations to serve from that partition, before moving to the next
partition (in a round-robin manner).
All of this is tied to the partition where the allocation was requested.
Each partition has a separate set of coefficients.
We might also treat the coefficients as probabilities, and use PRNG to
determine where to direct individual requests. But a PRNG seems fairly
expensive, and the budget approach works well.
We intentionally keep the "budget" fairly low, with the sum for a given
partition 100. That means we get to the same partition after only 100
allocations, keeping it more balanced. It wouldn't be hard to make the
budgets higher (e.g. matching the number of allocations per round), but
it might also make the behavior less smooth (long period of allocations
from each partition).
This is very simple/cheap, and over many allocations it has the same
effect. For periods of low activity it may diverge, but that does not
matter much (we care about high-activity periods much more).
---
.../pg_buffercache--1.7--1.8.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 43 +-
src/backend/storage/buffer/bufmgr.c | 3 +
src/backend/storage/buffer/freelist.c | 377 +++++++++++++++++-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 12 +-
6 files changed, 419 insertions(+), 22 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index f702a9db1a8..1834599c4b3 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -19,7 +19,10 @@ CREATE VIEW pg_buffercache_partitions AS
num_passes bigint, -- clocksweep passes
next_buffer integer, -- next victim buffer for clocksweep
total_allocs bigint, -- handled allocs (running total)
- num_allocs bigint); -- handled allocs (current cycle)
+ num_allocs bigint, -- handled allocs (current cycle)
+ total_req_allocs bigint, -- requested allocs (running total)
+ num_req_allocs bigint, -- handled allocs (current cycle)
+ weights int[]); -- balancing weights
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index aed0ecb59e9..81665209084 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,8 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -27,7 +29,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -844,6 +846,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
if (SRF_IS_FIRSTCALL())
{
+ TypeCacheEntry *typentry = lookup_type_cache(INT4OID, 0);
+
funcctx = SRF_FIRSTCALL_INIT();
/* Switch context when allocating stuff to be used in later calls */
@@ -873,6 +877,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "total_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "num_req_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "weigths",
+ typentry->typarray, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -893,11 +903,17 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
- uint64 buffer_total_allocs;
+ uint64 buffer_total_allocs,
+ buffer_total_req_allocs;
uint32 complete_passes,
next_victim_buffer,
- buffer_allocs;
+ buffer_allocs,
+ buffer_req_allocs;
+
+ int *weights;
+ Datum *dweights;
+ ArrayType *array;
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -906,8 +922,16 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
ClockSweepPartitionGetInfo(i,
- &complete_passes, &next_victim_buffer,
- &buffer_total_allocs, &buffer_allocs);
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs,
+ &buffer_total_req_allocs, &buffer_req_allocs,
+ &weights);
+
+ dweights = palloc_array(Datum, funcctx->max_calls);
+ for (int i = 0; i < funcctx->max_calls; i++)
+ dweights[i] = Int32GetDatum(weights[i]);
+
+ array = construct_array_builtin(dweights, funcctx->max_calls, INT4OID);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -933,6 +957,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffer_allocs);
nulls[7] = false;
+ values[8] = Int64GetDatum(buffer_total_req_allocs);
+ nulls[8] = false;
+
+ values[9] = Int64GetDatum(buffer_req_allocs);
+ nulls[9] = false;
+
+ values[10] = PointerGetDatum(array);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a3092ce801d..82c645a3b00 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3912,6 +3912,9 @@ BgBufferSync(WritebackContext *wb_context)
/* assume we can hibernate, any partition can set to false */
bool hibernate = true;
+ /* trigger partition rebalancing first */
+ StrategySyncBalance();
+
/* get the number of clocksweep partitions, and total alloc count */
StrategySyncPrepare(&num_partitions, &recent_alloc);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index d40b09f7e69..169071032b4 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -34,6 +34,23 @@
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * XXX needed for make ClockSweep fixed-size, should be tied to the number
+ * of buffer partitions (bufmgr.c already has MAX_CLOCKSWEEP_PARTITIONS, so
+ * at least set it to the same value).
+ */
+#define MAX_BUFFER_PARTITIONS 32
+
+/*
+ * Coefficient used to combine the old and new balance coefficients, using
+ * weighted average. The higher the value, the more the old value affects the
+ * result.
+ *
+ * XXX Doesn't this invalidate the interpretation as a probability to allocate
+ * from a given partition? Does it still sum to 100%?
+ */
+#define CLOCKSWEEP_HISTORY_COEFF 0.5
+
/*
* Information about one partition of the ClockSweep (on a subset of buffers).
*
@@ -66,9 +83,28 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /*
+ * Buffers that should have been allocated in this partition (but might
+ * have been redirected to keep allocations balanced).
+ */
+ pg_atomic_uint32 numRequestedAllocs;
+
/* running total of allocs */
pg_atomic_uint64 numTotalAllocs;
+ pg_atomic_uint64 numTotalRequestedAllocs;
+ /*
+ * Weights to balance buffer allocations for all the partitions. Each
+ * partition gets a vector of weights 0-100, determining what fraction
+ * of buffers to allocate from that particular. So [75, 15, 5, 5] would
+ * mean 75% allocations should go from partition 0, 15% from partition
+ * 1, and 5% from partitions 2&3. Each partition gets a different vector
+ * of weights.
+ *
+ * XXX Allocate a fixed-length array, to simplify working with array of
+ * the structs, etc.
+ */
+ uint8 balance[MAX_BUFFER_PARTITIONS];
} ClockSweep;
/*
@@ -130,7 +166,33 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
-static ClockSweep *ChooseClockSweep(void);
+static ClockSweep *ChooseClockSweep(bool balance);
+
+/*
+ * clocksweep allocation balancing
+ *
+ * To balance allocations from clocksweep partitions, each partition gets
+ * a set of "weights" determining the fraction of allocations to redirect
+ * to other partitions.
+ *
+ * We could do that based on a random number generator, but that seems too
+ * expensive. So instead we simply treat the probabilities as a budget, i.e.
+ * a number of allocations to serve from that partition, before moving to
+ * the next partition (in a round-robin manner).
+ *
+ * This is very simple/cheap, and over many allocations it has the same
+ * effect. For periods of low activity it may diverge, but that does not
+ * matter much (we care about high-activity periods much more).
+ *
+ * We intentionally keep the "budget" fairly low, with the sum for a given
+ * partition 100. That means we get to the same partition after only 100
+ * allocations, keeping it more balances. It wouldn't be hard to make the
+ * budgets higher (say, to match the expected number of allocations, i.e.
+ * about the average number of allocations from the past interval).
+ */
+static int clocksweep_partition_optimal = -1;
+static int clocksweep_partition = -1;
+static int clocksweep_count = 0;
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -142,7 +204,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
- ClockSweep *sweep = ChooseClockSweep();
+ ClockSweep *sweep = ChooseClockSweep(true);
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -233,11 +295,59 @@ calculate_partition_index(void)
* and that's cheaper. But how would that deal with odd number of nodes?
*/
static ClockSweep *
-ChooseClockSweep(void)
+ChooseClockSweep(bool balance)
{
- int index = calculate_partition_index();
+ /* What's the "optimal" partition? */
+ int index = calculate_partition_index();
+ ClockSweep *sweep = &StrategyControl->sweeps[index];
+
+ /*
+ * Did we migrate to a different core / NUMA node, affecting the
+ * clocksweep partition we should use? Switch to that partition.
+ */
+ if (clocksweep_partition_optimal != index)
+ {
+ clocksweep_partition_optimal = index;
+ clocksweep_partition = index;
+ clocksweep_count = sweep->balance[index];
+ }
+
+ /* we should have a valid partition */
+ Assert(clocksweep_partition_optimal != -1);
+ Assert(clocksweep_partition != -1);
+
+ /*
+ * If rebalancing is enabled, use the weights to redirect the allocations
+ * to match the desired distribution. We do that by using the partitions
+ * in a round-robin way, after allocating the "weight" of allocations
+ * from each partitions.
+ */
+ if (balance)
+ {
+ /*
+ * Ran out of allocations from the current partition? Move to the
+ * next partition with non-zero weight, and use the weight as a
+ * budget for allocations.
+ */
+ while (clocksweep_count == 0)
+ {
+ clocksweep_partition
+ = (clocksweep_partition + 1) % StrategyControl->num_partitions;
+
+ Assert((clocksweep_partition >= 0) &&
+ (clocksweep_partition < StrategyControl->num_partitions));
+
+ clocksweep_count = sweep->balance[clocksweep_partition];
+ }
- return &StrategyControl->sweeps[index];
+ /* account for the allocation - take it from the budget */
+ --clocksweep_count;
+
+ /* account for the alloc in the "optimal" (original) partition */
+ pg_atomic_fetch_add_u32(&sweep->numRequestedAllocs, 1);
+ }
+
+ return &StrategyControl->sweeps[clocksweep_partition];
}
/*
@@ -309,7 +419,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep(false)->numBufferAllocs, 1);
/*
* Use the "clock sweep" algorithm to find a free buffer
@@ -417,6 +527,224 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncBalance
+ * update partition weights, to balance the buffer allocations
+ *
+ * We want to give preference to allocating buffers on the same NUMA node,
+ * but that might lead to imbalance - a single process would only use a
+ * fraction of shared buffers. We don't want that, we want to utilize the
+ * whole shared buffers. The number of allocations in each partition may
+ * also change over time, so we need to adapt to that.
+ *
+ * To allow this "adaptive balancing", each partition has a set of weights,
+ * determining what fraction of allocations to direct to other partitions.
+ * For simplicity the coefficients are integers 0-100, expressing the
+ * percentage of allocations redirected to that partition.
+ *
+ * Consider for example weights [50, 25, 25, 0] for one of 4 partitions.
+ * This means 50% of allocations will be redirected to partition 0, 25%
+ * to partitions 1 and 2, and no allocations will go to partition 3.
+ *
+ * To calculate these weights, assume we know the number of allocations
+ * requested for each partition in the past interval. We can use this to
+ * calculate weights for the following interval, aiming to allocate the
+ * same (fair share) number of buffers from each partition.
+ *
+ * Note: This is based on number of allocations "originating" in a given
+ * partition. If an allocation is requested in a partition A, it's counted
+ * as allocation for A, even if it gets redirected to some other partition.
+ * The patch addes a new counter to track this.
+ *
+ * The main observation is that partitions get divided into two groups,
+ * depending on whether the number allocations is higher or lower than the
+ * target average. But the "total delta" for these two groups is the
+ * same, i.e. sum(abs(allocs - avg_allocs)) is the same. Therefore, the
+ * task is to "distribute" the excess allocations between the partitions
+ * with not enough allocations.
+ *
+ * Partitions with (nallocs < avg_nallocs) don't redirect any allocations.
+ *
+ * Partitions with (nallocs > avg_nallocs) redirect the extra allocations,
+ * with each target allocation getting a proportional part (with respect
+ * to the total delta).
+ *
+ * XXX In principle we might do without the new "requestedAllocs" counter,
+ * but we'd need to solve the matrix equation Ax=b, with [A,b] known
+ * (weights and allocs), and calculate x (requested allocs). But it's not
+ * quite clear this'd always have a solution.
+ */
+void
+StrategySyncBalance(void)
+{
+ /* snapshot of allocs for partitions */
+ uint32 allocs[MAX_BUFFER_PARTITIONS];
+
+ uint32 total_allocs = 0, /* total number of allocations */
+ avg_allocs, /* average allocations (per partition) */
+ delta_allocs = 0; /* sum of allocs above average */
+
+ /*
+ * Collect the number of allocations requested in the past interval.
+ * While at it, reset the counter to start the new interval.
+ *
+ * We lock the partitions one by one, so this is not exactly consistent
+ * snapshot of the counts, and the resets happen before we update the
+ * weights too. But we're only looking for heuristics anyway, so this
+ * should be good enough.
+ *
+ * A similar issue applies to the counter reset - we haven't updated
+ * the weights yet. Should be fine, we'll simply consider this in the
+ * next balancing cycle.
+ *
+ * XXX Does this need to worry about the completePasses too?
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ /* no need for a spinlock */
+ allocs[i] = pg_atomic_exchange_u32(&sweep->numRequestedAllocs, 0);
+
+ /* add the allocs to running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalRequestedAllocs, allocs[i]);
+
+ total_allocs += allocs[i];
+ }
+
+ /*
+ * Calculate the "fair share" of allocations per partition.
+ *
+ * XXX The last partition could be smaller, in which case it should be
+ * expected to handle fewer allocations. So this should be a weighted
+ * average. But for now a simple average is good enough.
+ */
+ avg_allocs = (total_allocs / StrategyControl->num_partitions);
+
+ /*
+ * Calculate the "delta" from balanced state, i.e. how many allocations
+ * we'd need to redistribute.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (allocs[i] > avg_allocs)
+ delta_allocs += (allocs[i] - avg_allocs);
+ }
+
+ /*
+ * Skip the rebalancing when there's not enough activity. In this case
+ * we just keep the current weights.
+ *
+ * XXX The threshold of 100 allocation is pretty arbitrary.
+ *
+ * XXX Maybe a better strategy would be to slowly return to the default
+ * weights, with each partition allocation only from itself?
+ *
+ * XXX Maybe we shouldn't even reset the counters in this case? But it
+ * should not matter, if the activity is low.
+ */
+ if (avg_allocs < 100)
+ {
+ elog(DEBUG1, "rebalance skipped: not enough allocations (allocs: %u)",
+ avg_allocs);
+ return;
+ }
+
+ /*
+ * Likewise, skip rebalancing if the misbalance is not significant. We
+ * consider it acceptable if the amount of allocations we'd need to
+ * redistribute is less than 10% of the average.
+ *
+ * XXX Again, these threshold are rather arbitrary.
+ */
+ if (delta_allocs < (avg_allocs * 0.1))
+ {
+ elog(DEBUG1, "rebalance skipped: delta within limit (delta: %u, threshold: %u)",
+ delta_allocs, (uint32) (avg_allocs * 0.1));
+ return;
+ }
+
+ /*
+ * Got to do the rebalancing. Go through the partitions, and for each
+ * partition decide if it gets to redirect or receive allocations.
+ *
+ * If a partition has fewer than average allocations, it won't redirect
+ * any allocations to other partitions. So it only has a single non-zero
+ * weight, and that's for itself.
+ *
+ * If a parttion has more than average allocations, it won't receive
+ * any redirected allocations. Instead, the excess allocations are
+ * redirected to the other partitions.
+ *
+ * The redistribution is "proportional" - if the excess allocations of
+ * a partition represent 10% of the "delta", then each partition that
+ * needs more allocations will get 10% of the gap from this one.
+ *
+ * XXX We should add hysteresis, to "dampen" the changes, and make
+ * sure it does not oscillate too much.
+ *
+ * XXX Ideally, the alternative partitions to use first would be the
+ * other partitions for the same node (if any).
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+ uint8 balance[MAX_BUFFER_PARTITIONS];
+
+ /* lock, we're going to modify the balance weights */
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+
+ /* reset the weights to start from scratch */
+ memset(balance, 0, sizeof(uint8) * MAX_BUFFER_PARTITIONS);
+
+ /* does this partition has fewer or more than avg_allocs? */
+ if (allocs[i] < avg_allocs)
+ {
+ /* fewer - don't redirect any allocations elsewhere */
+ balance[i] = 100;
+ }
+ else
+ {
+ /*
+ * more - redistribute the excess allocations
+ *
+ * Each "target" partition (with less than avg_allocs) should get
+ * a fraction proportional to (excess/delta) from this one.
+ */
+
+ /* fraction of the "total" delta */
+ double delta_frac = (allocs[i] - avg_allocs) * 1.0 / delta_allocs;
+
+ /* keep just enough allocations to meet the target */
+ balance[i] = (100.0 * avg_allocs / allocs[i]);
+
+ /* redirect the extra allocations */
+ for (int j = 0; j < StrategyControl->num_partitions; j++)
+ {
+ /* How many allocations to receive from i-th partition? */
+ uint32 receive_allocs = delta_frac * (avg_allocs - allocs[j]);
+
+ /* ignore partitions that don't need additional allocations */
+ if (allocs[j] > avg_allocs)
+ continue;
+
+ /* fraction to redirect */
+ balance[j] = (100.0 * receive_allocs / allocs[i]) + 0.5;
+ }
+ }
+
+ /* combine the old and new weights (hysteresis) */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ sweep->balance[j]
+ = CLOCKSWEEP_HISTORY_COEFF * sweep->balance[j] +
+ (1.0 - CLOCKSWEEP_HISTORY_COEFF) * balance[j];
+ }
+
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncPrepare -- prepare for sync of all partitions
*
@@ -443,6 +771,7 @@ StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
{
ClockSweep *sweep = &StrategyControl->sweeps[i];
+ /* XXX we don't need the spinlock to read atomics, no? */
SpinLockAcquire(&sweep->clock_sweep_lock);
if (num_buf_alloc)
{
@@ -627,7 +956,21 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->sweeps[i].completePasses = 0;
pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numRequestedAllocs, 0);
pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalRequestedAllocs, 0);
+
+ /*
+ * Initialize the weights - start by allocating 100% buffers from
+ * the current node / partition.
+ */
+ for (int j = 0; j < MAX_BUFFER_PARTITIONS; j++)
+ {
+ if (i == j)
+ StrategyControl->sweeps[i].balance[i] = 100;
+ else
+ StrategyControl->sweeps[i].balance[j] = 0;
+ }
}
/* No pending notification */
@@ -1001,8 +1344,10 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
void
ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes, uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs, uint32 *buffer_req_allocs,
+ int **weights)
{
ClockSweep *sweep = &StrategyControl->sweeps[idx];
@@ -1010,11 +1355,21 @@ ClockSweepPartitionGetInfo(int idx,
/* get the clocksweep stats */
*complete_passes = sweep->completePasses;
+
+ /* calculate the actual buffer ID */
*next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
- *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
*buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
- /* calculate the actual buffer ID */
- *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+ *buffer_total_req_allocs = pg_atomic_read_u64(&sweep->numTotalRequestedAllocs);
+ *buffer_req_allocs = pg_atomic_read_u32(&sweep->numRequestedAllocs);
+
+ /* return the weights in a newly allocated array */
+ *weights = palloc_array(int, StrategyControl->num_partitions);
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ (*weights)[i] = (int) sweep->balance[i];
+ }
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3307190f611..1118b386228 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,6 +508,7 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
+extern void StrategySyncBalance(void);
extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
extern int StrategySyncStart(int partition, uint32 *complete_passes,
int *first_buffer, int *num_buffers);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7052f9de57c..4e7b1fcd4ab 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -360,11 +360,13 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void ClockSweepPartitionGetInfo(int idx,
- uint32 *complete_passes,
- uint32 *next_victim_buffer,
- uint64 *buffer_total_allocs,
- uint32 *buffer_allocs);
-
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs,
+ uint64 *buffer_total_req_allocs,
+ uint32 *buffer_req_allocs,
+ int **weights);
/* inline functions */
--
2.51.1
[text/x-patch] v20251126-0002-clock-sweep-basic-partitioning.patch (33.4K, ../[email protected]/9-v20251126-0002-clock-sweep-basic-partitioning.patch)
download | inline diff:
From 670ef95cfbec187fbc3e2cce641dec9bd79a092e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 11 Nov 2025 12:03:32 +0100
Subject: [PATCH v20251126 2/9] clock-sweep: basic partitioning
Partitions the "clock-sweep" algorithm to work on individual partitions,
one by one. Each backend process is mapped to one "home" partition, with
an independent clock hand. This reduces contention for workloads with
significant buffer pressure.
The patch extends the "pg_buffercache_partitions" view to include
information about the clock-sweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
Note: There's a problem with some tests running out of unpinned buffers,
due to (intentionally) setting shared buffers very low. That happens
because StrategyGetBuffer() only searches a single partition, and it
has a couple more issues.
---
.../pg_buffercache--1.7--1.8.sql | 8 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 32 +-
src/backend/storage/buffer/buf_init.c | 8 +
src/backend/storage/buffer/bufmgr.c | 186 ++++++++----
src/backend/storage/buffer/freelist.c | 283 +++++++++++++++---
src/include/storage/buf_internals.h | 5 +-
src/include/storage/bufmgr.h | 5 +
src/test/recovery/t/027_stream_regress.pl | 5 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 430 insertions(+), 103 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
index d62b8339bfc..f702a9db1a8 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -13,7 +13,13 @@ CREATE VIEW pg_buffercache_partitions AS
(partition integer, -- partition index
num_buffers integer, -- number of buffers in the partition
first_buffer integer, -- first buffer of partition
- last_buffer integer); -- last buffer of partition
+ last_buffer integer, -- last buffer of partition
+
+ -- clocksweep counters
+ num_passes bigint, -- clocksweep passes
+ next_buffer integer, -- next victim buffer for clocksweep
+ total_allocs bigint, -- handled allocs (running total)
+ num_allocs bigint); -- handled allocs (current cycle)
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8c89855192f..aed0ecb59e9 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -865,6 +865,14 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "num_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "next_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "total_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "num_allocs",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -885,12 +893,22 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffer_total_allocs;
+
+ uint32 complete_passes,
+ next_victim_buffer,
+ buffer_allocs;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &num_buffers,
&first_buffer, &last_buffer);
+ ClockSweepPartitionGetInfo(i,
+ &complete_passes, &next_victim_buffer,
+ &buffer_total_allocs, &buffer_allocs);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -903,6 +921,18 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[3] = Int32GetDatum(last_buffer);
nulls[3] = false;
+ values[4] = Int64GetDatum(complete_passes);
+ nulls[4] = false;
+
+ values[5] = Int32GetDatum(next_victim_buffer);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffer_total_allocs);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffer_allocs);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 528a368a8b7..0362fda24aa 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -319,3 +319,11 @@ BufferPartitionGet(int idx, int *num_buffers,
elog(ERROR, "invalid partition index");
}
+
+/* return parameters before the partitions are initialized (during sizing) */
+void
+BufferPartitionParams(int *num_partitions)
+{
+ if (num_partitions)
+ *num_partitions = NUM_CLOCK_SWEEP_PARTITIONS;
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 327ddb7adc8..a3092ce801d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3608,33 +3608,29 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
*
- * This is called periodically by the background writer process.
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
*
- * Returns true if it's appropriate for the bgwriter process to go into
- * low-power hibernation mode. (This happens if the strategy clock-sweep
- * has been "lapped" and no buffer allocations have occurred recently,
- * or if the bgwriter has been effectively disabled by setting
- * bgwriter_lru_maxpages to 0.)
+ * XXX might be better to have a per-partition struct with all the info
*/
-bool
-BgBufferSync(WritebackContext *wb_context)
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
+static bool
+BgBufferSyncPartition(WritebackContext *wb_context, int num_partitions,
+ int partition, int recent_alloc_partition)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
-
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
/* Moving averages of allocation rate and clean-buffer density */
static float smoothed_alloc = 0;
@@ -3662,25 +3658,16 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
+
/*
* Find out where the clock-sweep currently is, and how many buffer
* allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
-
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.buf_alloc += recent_alloc;
-
- /*
- * If we're not running the LRU scan, just stop after doing the stats
- * stuff. We mark the saved state invalid so that we can recover sanely
- * if LRU scan is turned back on later.
- */
- if (bgwriter_lru_maxpages <= 0)
- {
- saved_info_valid = false;
- return true;
- }
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
/*
* Compute strategy_delta = how many buffers have been scanned by the
@@ -3692,17 +3679,17 @@ BgBufferSync(WritebackContext *wb_context)
*/
if (saved_info_valid)
{
- int32 passes_delta = strategy_passes - prev_strategy_passes;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
Assert(strategy_delta >= 0);
- if ((int32) (next_passes - strategy_passes) > 0)
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
{
/* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3710,11 +3697,11 @@ BgBufferSync(WritebackContext *wb_context)
strategy_delta, bufs_to_lap);
#endif
}
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
{
/* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
next_passes, next_to_clean,
@@ -3734,9 +3721,9 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id,
strategy_delta);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
}
else
@@ -3750,15 +3737,16 @@ BgBufferSync(WritebackContext *wb_context)
strategy_passes, strategy_buf_id);
#endif
strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
/* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* XXX this needs to happen only after all partitions */
+ /* saved_info_valid = true; */
/*
* Compute how many buffers had to be scanned for each new allocation, ie,
@@ -3766,9 +3754,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -3778,7 +3766,7 @@ BgBufferSync(WritebackContext *wb_context)
* strategy point and where we've scanned ahead to, based on the smoothed
* density estimate.
*/
- bufs_ahead = NBuffers - bufs_to_lap;
+ bufs_ahead = num_buffers - bufs_to_lap;
reusable_buffers_est = (float) bufs_ahead / smoothed_density;
/*
@@ -3786,10 +3774,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -3816,7 +3804,7 @@ BgBufferSync(WritebackContext *wb_context)
* the BGW will be called during the scan_whole_pool time; slice the
* buffer pool into that many sections.
*/
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
{
@@ -3841,20 +3829,20 @@ BgBufferSync(WritebackContext *wb_context)
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
wb_context);
- if (++next_to_clean >= NBuffers)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- next_to_clean = 0;
- next_passes++;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
}
num_to_scan--;
if (sync_state & BUF_WRITTEN)
{
reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
{
PendingBgWriterStats.maxwritten_clean++;
break;
@@ -3868,7 +3856,7 @@ BgBufferSync(WritebackContext *wb_context)
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
smoothed_density, reusable_buffers_est, upcoming_alloc_est,
bufs_to_lap - num_to_scan,
num_written,
@@ -3898,8 +3886,74 @@ BgBufferSync(WritebackContext *wb_context)
#endif
}
+ /* can this partition hibernate */
+ return (bufs_to_lap == 0 && recent_alloc_partition == 0);
+}
+
+/*
+ * BgBufferSync -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode. (This happens if the strategy clock-sweep
+ * has been "lapped" and no buffer allocations have occurred recently,
+ * or if the bgwriter has been effectively disabled by setting
+ * bgwriter_lru_maxpages to 0.)
+ */
+bool
+BgBufferSync(WritebackContext *wb_context)
+{
+ /* info obtained from freelist.c */
+ uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
+
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
+
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
+
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
+
+ /* Report buffer alloc counts to pgstat */
+ PendingBgWriterStats.buf_alloc += recent_alloc;
+
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
+ /*
+ * If we're not running the LRU scan, just stop after doing the stats
+ * stuff. We mark the saved state invalid so that we can recover sanely
+ * if LRU scan is turned back on later.
+ */
+ if (bgwriter_lru_maxpages <= 0)
+ {
+ saved_info_valid = false;
+ return true;
+ }
+
+ /*
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* hibernate if all partitions can hibernate */
+ hibernate &= BgBufferSyncPartition(wb_context, num_partitions,
+ partition, recent_alloc_partition);
+ }
+
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 28d952b3534..d40b09f7e69 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,27 +15,47 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <sched.h>
+#endif
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* clock-sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -46,11 +66,32 @@ typedef struct
uint32 completePasses; /* Complete cycles of the clock-sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+ /* running total of allocs */
+ pg_atomic_uint64 numTotalAllocs;
+
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
+
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
*/
int bgwprocno;
+ // the _attribute_ does not work on Windows, it seems
+ //int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_partitions;
+
+ /* clocksweep partitions */
+ ClockSweep sweeps[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -89,6 +130,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -100,6 +142,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -107,14 +150,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -140,19 +183,61 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
+}
+
+/*
+ * calculate_partition_index
+ * calculate the buffer / clock-sweep partition to use
+ *
+ * use PID to determine the buffer partition
+ *
+ * XXX We could use NUMA node / core ID to pick partition, but we'd need
+ * to handle cases with fewer nodes/cores than partitions somehow. Although,
+ * maybe the balancing would handle that too.
+ */
+static int
+calculate_partition_index(void)
+{
+ return (MyProcPid % StrategyControl->num_partitions);
+}
+
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
}
/*
@@ -224,9 +309,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
- /* Use the "clock sweep" algorithm to find a free buffer */
+ /*
+ * Use the "clock sweep" algorithm to find a free buffer
+ *
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
+ *
+ * XXX This only searches a single partition, which can result in "no
+ * unpinned buffers available" even if there are buffers in other
+ * partitions. Should be fixed by falling back to other partitions if
+ * needed.
+ *
+ * XXX Also, the trycounter should not be set to NBuffers, but to buffer
+ * count for that one partition. In fact, this should not call ClockSweepTick
+ * for every iteration. The call is likely quite expensive (does a lot
+ * of stuff), and also may return a different partition on each call.
+ * We should just do it once, and then do the for(;;) loop. And then
+ * maybe advance to the next partition, until we scan through all of them.
+ */
trycounter = NBuffers;
for (;;)
{
@@ -306,6 +417,46 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
+/*
+ * StrategySyncPrepare -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ uint32 allocs = pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+
+ /* include the count in the running total */
+ pg_atomic_fetch_add_u64(&sweep->numTotalAllocs, allocs);
+
+ *num_buf_alloc += allocs;
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -313,37 +464,44 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -380,6 +538,9 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+
+ BufferPartitionParams(&num_partitions);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
@@ -387,6 +548,10 @@ StrategyShmemSize(void)
/* size of the shared replacement strategy control block */
size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -402,6 +567,10 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_partitions;
+
+ num_partitions = BufferPartitionCount();
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -419,7 +588,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, sweeps)) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -431,15 +601,40 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock-sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ pg_atomic_init_u64(&StrategyControl->sweeps[i].numTotalAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
}
else
Assert(!init);
@@ -803,3 +998,23 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes, uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs, uint32 *buffer_allocs)
+{
+ ClockSweep *sweep = &StrategyControl->sweeps[idx];
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ /* get the clocksweep stats */
+ *complete_passes = sweep->completePasses;
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *buffer_total_allocs = pg_atomic_read_u64(&sweep->numTotalAllocs);
+
+ /* calculate the actual buffer ID */
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 139055a4a7d..3307190f611 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -508,7 +508,9 @@ extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
@@ -554,5 +556,6 @@ extern int BufferPartitionCount(void);
extern int BufferPartitionNodes(void);
extern void BufferPartitionGet(int idx, int *num_buffers,
int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions);
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 24860c6c2c4..7052f9de57c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -359,6 +359,11 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
+extern void ClockSweepPartitionGetInfo(int idx,
+ uint32 *complete_passes,
+ uint32 *next_victim_buffer,
+ uint64 *buffer_total_allocs,
+ uint32 *buffer_allocs);
/* inline functions */
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 589c79d97d3..98b146ed4b7 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -18,6 +18,11 @@ $node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf',
'max_prepared_transactions = 10');
+# The default is 1MB, which is not enough with clock-sweep partitioning.
+# Increase to 32MB, so that we don't get "no unpinned buffers".
+$node_primary->append_conf('postgresql.conf',
+ 'shared_buffers = 32MB');
+
# Enable pg_stat_statements to force tests to do query jumbling.
# pg_stat_statements.max should be large enough to hold all the entries
# of the regression database.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d3edff346a8..241f175e9da 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.51.1
[text/x-patch] v20251126-0001-Infrastructure-for-partitioning-shared-buf.patch (13.6K, ../[email protected]/10-v20251126-0001-Infrastructure-for-partitioning-shared-buf.patch)
download | inline diff:
From abe7cf9f1314bdaae2361ad237ef859e00b69c07 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Wed, 17 Sep 2025 23:04:29 +0200
Subject: [PATCH v20251126 1/9] Infrastructure for partitioning shared buffers
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, etc. This serves as a source of truth
for later patches (e.g. to partition clock-sweep).
The registry is a small BufferPartitions array in shared memory, with
partitions sized to be a fair share of shared buffers. Later patches may
improve this to consider NUMA, and similar details.
With the feature disabled (GUC set to empty list), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* Maybe the number of partitions should be configurable? Right now it's
hard-coded as 4, but testing shows increasing to e.g. 16) can be
beneficial.
* This partitioning is independent of the partitions defined in
lwlock.h, which defines 128 partitions to reduce lock conflict on the
buffer mapping hashtable. The number of partitions introduced by this
patch is expected to be much lower (a dozen or so).
---
.../pg_buffercache--1.7--1.8.sql | 23 +++
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 86 +++++++++++
src/backend/storage/buffer/buf_init.c | 144 +++++++++++++++++-
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 19 +++
src/tools/pgindent/typedefs.list | 2 +
7 files changed, 280 insertions(+), 2 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
diff --git a/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
new file mode 100644
index 00000000000..d62b8339bfc
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.7--1.8.sql
@@ -0,0 +1,23 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.8'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, -- partition index
+ num_buffers integer, -- number of buffers in the partition
+ first_buffer integer, -- first buffer of partition
+ last_buffer integer); -- last buffer of partition
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index 11499550945..d2fa8ba53ba 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.7'
+default_version = '1.8'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae1712fc93c..8c89855192f 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_OS_PAGES_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 4
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -101,6 +102,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -826,3 +828,87 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(num_buffers);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(first_buffer);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(last_buffer);
+ nulls[3] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 6fd3a6bbac5..528a368a8b7 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -17,6 +17,11 @@
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +29,14 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+/* *
+ * number of buffer partitions */
+#define NUM_CLOCK_SWEEP_PARTITIONS 4
+
+/* Array of structs with information about buffer ranges */
+BufferPartitions *BufferPartitionsArray = NULL;
+
+static void buffer_partitions_init(void);
/*
* Data Structures:
@@ -70,7 +83,15 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS),
+ &foundParts);
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -112,6 +133,9 @@ BufferManagerShmemInit(void)
{
int i;
+ /* Initialize buffer partitions (calculate buffer ranges). */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -175,5 +199,123 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), NUM_CLOCK_SWEEP_PARTITIONS)));
+
return size;
}
+
+/*
+ * Sanity checks of buffers partitions - there must be no gaps, it must cover
+ * the whole range of buffers, etc.
+ */
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ Assert(BufferPartitionsArray->npartitions > 0);
+
+ for (int i = 0; i < BufferPartitionsArray->npartitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (BufferPartitionsArray->npartitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+/*
+ * buffer_partitions_init
+ * Initialize array of buffer partitions.
+ */
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+
+ /* number of buffers per partition (make sure to not overflow) */
+ int part_buffers
+ = ((int64) NBuffers + (NUM_CLOCK_SWEEP_PARTITIONS - 1)) / NUM_CLOCK_SWEEP_PARTITIONS;
+
+ BufferPartitionsArray->npartitions = NUM_CLOCK_SWEEP_PARTITIONS;
+
+ for (int n = 0; n < BufferPartitionsArray->npartitions; n++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[n];
+
+ /* buffers this partition should get (last partition can get fewer) */
+ int num_buffers = Min(remaining_buffers, part_buffers);
+
+ remaining_buffers -= num_buffers;
+
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ buffer += num_buffers;
+ }
+
+ AssertCheckBufferPartitions();
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5400c56a965..139055a4a7d 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -345,6 +345,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -549,4 +550,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern int BufferPartitionNodes(void);
+extern void BufferPartitionGet(int idx, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b5f8f3c5d42..24860c6c2c4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -153,6 +153,25 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+/*
+ * information about one partition of shared buffers
+ *
+ * first/last buffer - the values are inclusive
+ */
+typedef struct BufferPartition
+{
+ int num_buffers; /* number of buffers */
+ int first_buffer; /* first buffer of partition */
+ int last_buffer; /* last buffer of partition */
+} BufferPartition;
+
+/* an array of information about all partitions */
+typedef struct BufferPartitions
+{
+ int npartitions; /* number of partitions */
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* to avoid having to expose buf_internals.h here */
typedef struct WritebackContext WritebackContext;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dfcd619bfee..d3edff346a8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.51.1
view thread (89+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox