public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Jakub Wartak <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Adding basic NUMA awareness
Date: Mon, 28 Jul 2025 16:19:07 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAKZiRmy7VW5b0OKwHXtRSudCBFB48-kYnOEXeqrLfbTQ=5qQrA@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAKZiRmxemXXVAouzM4Ls7pA7e0u6CVSLJeL_phKkmGPOzvUv_g@mail.gmail.com>
<[email protected]>
Hi,
Here's a somewhat cleaned up v3 of this patch series, with various
improvements and a lot of cleanup. Still WIP, but I hope it resolves the
various crashes reported for v2, but it still requires --with-libnuma
(it won't build without it).
I'm aware there's an ongoing discussion about removing the freelists,
and changing the clocksweep in some way. If that happens, the relevant
parts of this series will need some adjustment, of course. I haven't
looked into that yet, I plan to review those patches soon.
main changes in v3
------------------
1) I've introduced "registry" of the buffer partitions (imagine a small
array of structs), serving as a source of truth for places that need
info about the partitions (range of buffers, ...).
With v2 there was no "shared definition" - the shared buffers, freelist
and clocksweep did their own thing. But per the discussion it doesn't
really make much sense for to partition buffers in different ways.
So in v3 the 0001 patch defines the partitions, records them in shared
memory (in a small array), and the later parts just reuse this.
I also added a pg_buffercache_partitions() listing the partitions, with
first/last buffer, etc. The freelist/clocksweep patches add additional
information.
2) The PGPROC part introduces a similar registry, even though there are
no other patches building on this. But it seemed useful to have a clear
place recording this info.
There's also a view pg_buffercache_pgproc. The pg_buffercache location
is a bit bogus - it has nothing to do with buffers, but it was good
enough for now.
3) The PGPROC partitioning is reworked and should fix the crash with the
GUC set to "off".
4) This still doesn't do anything about "balancing" the clocksweep. I
have some ideas how to do that, I'll work on that next.
simple benchmark
----------------
I did a simple benchmark, measuring pgbench throughput with scale still
fitting into RAM, but much larger (~2x) than shared buffers. See the
attached test script, testing builds with more and more of the patches.
I'm attaching results from two different machines (the "usual" 2P xeon
and also a much larger cloud instance with EPYC/Genoa) - both the raw
CSV files, with average tps and percentiles, and PDFs. The PDFs also
have a comparison either to the "preceding" build (right side), or to
master (below the table).
There's results for the three "pgbench pinning" strategies, and that can
have pretty significant impact (colocated generally performs much better
than either "none" or "random").
For the "bigger" machine (wiuth 176 cores) the incremental results look
like this (for pinning=none, i.e. regular pgbench):
mode s_b buffers localal no-tail freelist sweep pgproc pinning
====================================================================
prepared 16GB 99% 101% 100% 103% 111% 99% 102%
32GB 98% 102% 99% 103% 107% 101% 112%
8GB 97% 102% 100% 102% 101% 101% 106%
--------------------------------------------------------------------
simple 16GB 100% 100% 99% 105% 108% 99% 108%
32GB 98% 101% 100% 103% 100% 101% 97%
8GB 100% 100% 101% 99% 100% 104% 104%
The way I read this is that the first three patches have about no impact
on throughput. Then freelist partitioning and (especially) clocksweep
partitioning can help quite a bit. pgproc is again close to ~0%, and
PGPROC pinning can help again (but this part is merely experimental).
For the xeon the differences (in either direction) are much smaller, so
I'm not going to post it here. It's in the PDF, though.
I think this looks reasonable. The way I see this patch series is not
about improving peak throughput, but more about reducing imbalance and
making the behavior more consistent.
The results are more a confirmation there's not some sort of massive
overhead, somewhere. But I'll get to this in a minute.
To quantify this kind of improvement, I think we'll need tests that
intentionally cause (or try to) imbalance. If you have ideas for such
tests, let me know.
overhead of partitioning calculation
------------------------------------
Regarding the "overhead", while the results look mostly OK, I think
we'll need to rethink the partitioning scheme - particularly how the
partition size is calculated. The current scheme has to use %, which can
be somewhat expensive.
The 0001 patch calculates a "chunk size", which is the smallest number
of buffers it can "assign" to a NUMA node. This depends on how many
buffer descriptors fit onto a single memory page, and it's either 512KB
(with 4KB pages), or 256MB (with 2MB huge pages). And then each NUMA
node gets multiple chunks, to cover shared_buffers/num_nodes. But this
can be an arbitrary number - it minimizes the imbalance, but it also
forces the use of % and / in the formulas.
AFAIK if we required the partitions to be 2^k multiples of the chunk
size, we could switch to using shifts and masking. Which is supposed to
be much faster. But I haven't measured this, and the cost is that some
of the nodes could get much less memory. Maybe that's fine.
reserving number of huge pages
------------------------------
The other thing I realized is that partitioning buffers with huge pages
is quite tricky, and can easily lead to SIGBUS when accessing the memory
later. The crashes I saw happen like this:
1) figure # of pages needed (using shared_memory_size_in_huge_pages)
This can be 16828 for shared_buffers=32GB.
2) make sure there's enough huge pages
echo 16828 > /proc/sys/vm/nr_hugepages
3) start postgres - everything seems to works just fine
4) query pg_buffercache_numa - triggers SIGBUS accessing memory for a
valid buffer (usually ~2GB from the end)
It took me ages to realize what's happening, but it's very simple. The
nr_hugepages is a global limit, but it's also translated into limits for
each NUMA node. So when you write 16828 to it, in a 4-node system each
node gets 1/4 of that. See
$ numastat -cm
Then we do the mmap(), and everything looks great, because there really
is enough huge pages and the system can allocate memory from any NUMA
node it needs.
And then we come around, and do the numa_tonode_memory(). And that's
where the issues start, because AFAIK this does not check the per-node
limit of huge pages in any way. It just appears to work. And then later,
when we finally touch the buffer, it tries to actually allocate the
memory on the node, and realizes there's not enough huge pages. And
triggers the SIGBUS.
You may ask why the per-node limit is too low. We still need just
shared_memory_size_in_huge_pages, right? And if we were partitioning the
whole memory segment, that'd be true. But we only to that for shared
buffers, and there's a lot of other shared memory - could be 1-2GB or
so, depending on the configuration.
And this gets placed on one of the nodes, and it counts against the
limit on that particular node. And so it doesn't have enough huge pages
to back the partition of shared buffers.
The only way around this I found is by inflating the number of huge
pages, significantly above the shared_memory_size_in_huge_pages value.
Just to make sure the nodes get enough huge pages.
I don't know what to do about this. It's quite annoying. If we only used
huge pages for the partitioned parts, this wouldn't be a problem.
I also realize this can be used to make sure the memory is balanced on
NUMA systems. Because if you set nr_hugepages, the kernel will ensure
the shared memory is distributed on all the nodes.
It won't have the benefits of "coordinating" the buffers and buffer
descriptors, and so on. But it will be balanced.
regards
--
Tomas Vondra
Attachments:
[text/x-patch] v3-0007-NUMA-pin-backends-to-NUMA-nodes.patch (3.5K, ../[email protected]/2-v3-0007-NUMA-pin-backends-to-NUMA-nodes.patch)
download | inline diff:
From 3b3a929b007205c46b3c193cf38e3edf71084af7 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 27 May 2025 23:08:48 +0200
Subject: [PATCH v3 7/7] NUMA: pin backends to NUMA nodes
When initializing the backend, we pick a PGPROC entry from the right
NUMA node where the backend is running. But the process can move to a
different core / node, so to prevent that we pin it.
---
src/backend/storage/lmgr/proc.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 11259151a7d..dbb4cbb1bfa 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -766,6 +766,27 @@ InitProcess(void)
}
MyProcNumber = GetNumberFromPGProc(MyProc);
+ /*
+ * Optionally, restrict the process to only run on CPUs from the same NUMA
+ * as the PGPROC. We do this even if the PGPROC has a different NUMA node,
+ * but not for PGPROC entries without a node (i.e. aux/2PC entries).
+ *
+ * This also means we only do this with numa_procs_interleave, because
+ * without that we'll have numa_node=-1 for all PGPROC entries.
+ *
+ * FIXME add proper error-checking for libnuma functions
+ */
+ if (numa_procs_pin && MyProc->numa_node != -1)
+ {
+ struct bitmask *cpumask = numa_allocate_cpumask();
+
+ numa_node_to_cpus(MyProc->numa_node, cpumask);
+
+ numa_sched_setaffinity(MyProcPid, cpumask);
+
+ numa_free_cpumask(cpumask);
+ }
+
/*
* Cross-check that the PGPROC is of the type we expect; if this were not
* the case, it would get returned to the wrong list.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ee4684d1b8..3f88659b49f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -150,6 +150,7 @@ bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
bool numa_procs_interleave = false;
+bool numa_procs_pin = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7b718760248..862341e137e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2156,6 +2156,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_pin", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables pinning backends to NUMA nodes (matching the PGPROC node)."),
+ gettext_noop("When enabled, sets affinity to CPUs from the same NUMA node."),
+ },
+ &numa_procs_pin,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index cdeee8dccba..a97741c6707 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -182,6 +182,7 @@ extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT bool numa_procs_interleave;
+extern PGDLLIMPORT bool numa_procs_pin;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v3-0006-NUMA-interleave-PGPROC-entries.patch (46.8K, ../[email protected]/3-v3-0006-NUMA-interleave-PGPROC-entries.patch)
download | inline diff:
From 5addb5973ce571debebf07b17adc07eb828a48ee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:39:08 +0200
Subject: [PATCH v3 6/7] NUMA: interleave PGPROC entries
The goal is to distribute ProcArray (or rather PGPROC entries and
associated fast-path arrays) to NUMA nodes.
We can't do this by simply interleaving pages, because that wouldn't
work for both parts at the same time. We want to place the PGPROC and
it's fast-path locking structs on the same node, but the structs are
of different sizes, etc.
Another problem is that PGPROC entries are fairly small, so with huge
pages and reasonable values of max_connections everything fits onto a
single page. We don't want to make this incompatible with huge pages.
Note: If we eventually switch to allocating separate shared segments for
different parts (to allow on-line resizing), we could keep using regular
pages for procarray, and this would not be such an issue.
To make this work, we split the PGPROC array into per-node segments,
each with about (MaxBackends / numa_nodes) entries, and one segment for
auxiliary processes and prepared transations. And we do the same thing
for fast-path arrays.
The PGPROC segments are laid out like this (e.g. for 2 NUMA nodes):
- PGPROC array / node #0
- PGPROC array / node #1
- PGPROC array / aux processes + 2PC transactions
- fast-path arrays / node #0
- fast-path arrays / node #1
- fast-path arrays / aux processes + 2PC transaction
Each segment is aligned to (starts at) memory page, and is effectively a
multiple of multiple memory pages.
Having a single PGPROC array made certain operations easiers - e.g. it
was possible to iterate the array, and GetNumberFromPGProc() could
calculate offset by simply subtracting PGPROC pointers. With multiple
segments that's not possible, but the fallout is minimal.
Most places accessed PGPROC through PROC_HDR->allProcs, and can continue
to do so, except that now they get a pointer to the PGPROC (which most
places wanted anyway).
With the feature disabled, there's only a single "partition" for all
PGPROC entries.
Similarly to the buffer partitioning, this introduces a small "registry"
of partitions, as a source of truth. And then also a new "system" view
"pg_buffercache_pgproc" showing basic infromation abouut the partitions.
Note: There's an indirection, though. But the pointer does not change,
so hopefully that's not an issue. And each PGPROC entry gets an explicit
procnumber field, which is the index in allProcs, GetNumberFromPGProc
can simply return that.
Each PGPROC also gets numa_node, tracking the NUMA node, so that we
don't have to recalculate that. This is used by InitProcess() to pick
a PGPROC entry from the local NUMA node.
Note: The scheduler may migrate the process to a different CPU/node
later. Maybe we should consider pinning the process to the node?
---
.../pg_buffercache--1.6--1.7.sql | 19 +
contrib/pg_buffercache/pg_buffercache_pages.c | 94 +++
src/backend/access/transam/clog.c | 4 +-
src/backend/postmaster/pgarch.c | 2 +-
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/storage/buffer/buf_init.c | 2 -
src/backend/storage/buffer/freelist.c | 2 +-
src/backend/storage/ipc/procarray.c | 63 +-
src/backend/storage/lmgr/lock.c | 6 +-
src/backend/storage/lmgr/proc.c | 565 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/proc.h | 14 +-
src/tools/pgindent/typedefs.list | 1 +
15 files changed, 722 insertions(+), 64 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index b7d8ea45ed7..c48950a9d3b 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -23,3 +23,22 @@ REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_pgproc()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_pgproc'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_pgproc AS
+ SELECT P.* FROM pg_buffercache_pgproc() AS P
+ (partition integer,
+ numa_node integer, num_procs integer, pgproc_ptr bigint, fastpath_ptr bigint);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_pgproc() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_pgproc FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_pgproc() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_pgproc TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 5169655ae78..22396f36c09 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -15,6 +15,7 @@
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/proc.h"
#include "utils/rel.h"
@@ -28,6 +29,7 @@
#define NUM_BUFFERCACHE_NUMA_ELEM 3
#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
+#define NUM_BUFFERCACHE_PGPROC_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -102,6 +104,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
+PG_FUNCTION_INFO_V1(pg_buffercache_pgproc);
/* Only need to touch memory once per backend process lifetime */
@@ -905,3 +908,94 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
else
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Inquire about partitioning of PGPROC array.
+ */
+Datum
+pg_buffercache_pgproc(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PGPROC_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_procs",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "pgproc_ptr",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "fastpath_ptr",
+ INT8OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = ProcPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_procs;
+
+ void *pgproc_ptr,
+ *fastpath_ptr;
+
+ Datum values[NUM_BUFFERCACHE_PGPROC_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PGPROC_ELEM];
+
+ ProcPartitionGet(i, &numa_node, &num_procs,
+ &pgproc_ptr, &fastpath_ptr);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_procs);
+ nulls[2] = false;
+
+ values[3] = PointerGetDatum(pgproc_ptr);
+ nulls[3] = false;
+
+ values[4] = PointerGetDatum(fastpath_ptr);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index e80fbe109cf..928d126d0ee 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -574,7 +574,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Walk the list and update the status of all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &ProcGlobal->allProcs[nextidx];
+ PGPROC *nextproc = ProcGlobal->allProcs[nextidx];
int64 thispageno = nextproc->clogGroupMemberPage;
/*
@@ -633,7 +633,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *wakeproc = &ProcGlobal->allProcs[wakeidx];
+ PGPROC *wakeproc = ProcGlobal->allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&wakeproc->clogGroupNext);
pg_atomic_write_u32(&wakeproc->clogGroupNext, INVALID_PROC_NUMBER);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..e28e0f7d3bd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -289,7 +289,7 @@ PgArchWakeup(void)
* be relaunched shortly and will start archiving.
*/
if (arch_pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[arch_pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[arch_pgprocno]->procLatch);
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..087279a6a8e 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -649,7 +649,7 @@ WakeupWalSummarizer(void)
LWLockRelease(WALSummarizerLock);
if (pgprocno != INVALID_PROC_NUMBER)
- SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[pgprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 5b65a855b29..fb52039e1a6 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -500,8 +500,6 @@ buffer_partitions_prepare(void)
if (numa_nodes < 1)
numa_nodes = 1;
- elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
-
/*
* XXX A bit weird. Do we need to worry about postmaster? Could this even
* run outside postmaster? I don't think so.
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index ff02dc8e00b..d8d602f0a4e 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -416,7 +416,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* actually fine because procLatch isn't ever freed, so we just can
* potentially set the wrong process' (or no process') latch.
*/
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ SetLatch(&ProcGlobal->allProcs[bgwprocno]->procLatch);
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index bf987aed8d3..3e86e4ca2ae 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -268,7 +268,7 @@ typedef enum KAXCompressReason
static ProcArrayStruct *procArray;
-static PGPROC *allProcs;
+static PGPROC **allProcs;
/*
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
@@ -502,7 +502,7 @@ ProcArrayAdd(PGPROC *proc)
int this_procno = arrayP->pgprocnos[index];
Assert(this_procno >= 0 && this_procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[this_procno].pgxactoff == index);
+ Assert(allProcs[this_procno]->pgxactoff == index);
/* If we have found our right position in the array, break */
if (this_procno > pgprocno)
@@ -538,9 +538,9 @@ ProcArrayAdd(PGPROC *proc)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff == index - 1);
+ Assert(allProcs[procno]->pgxactoff == index - 1);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -581,7 +581,7 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
myoff = proc->pgxactoff;
Assert(myoff >= 0 && myoff < arrayP->numProcs);
- Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]].pgxactoff == myoff);
+ Assert(ProcGlobal->allProcs[arrayP->pgprocnos[myoff]]->pgxactoff == myoff);
if (TransactionIdIsValid(latestXid))
{
@@ -636,9 +636,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid)
int procno = arrayP->pgprocnos[index];
Assert(procno >= 0 && procno < (arrayP->maxProcs + NUM_AUXILIARY_PROCS));
- Assert(allProcs[procno].pgxactoff - 1 == index);
+ Assert(allProcs[procno]->pgxactoff - 1 == index);
- allProcs[procno].pgxactoff = index;
+ allProcs[procno]->pgxactoff = index;
}
/*
@@ -860,7 +860,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
/* Walk the list and clear all XIDs. */
while (nextidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[nextidx];
+ PGPROC *nextproc = allProcs[nextidx];
ProcArrayEndTransactionInternal(nextproc, nextproc->procArrayGroupMemberXid);
@@ -880,7 +880,7 @@ ProcArrayGroupClearXid(PGPROC *proc, TransactionId latestXid)
*/
while (wakeidx != INVALID_PROC_NUMBER)
{
- PGPROC *nextproc = &allProcs[wakeidx];
+ PGPROC *nextproc = allProcs[wakeidx];
wakeidx = pg_atomic_read_u32(&nextproc->procArrayGroupNext);
pg_atomic_write_u32(&nextproc->procArrayGroupNext, INVALID_PROC_NUMBER);
@@ -1526,7 +1526,7 @@ TransactionIdIsInProgress(TransactionId xid)
pxids = other_subxidstates[pgxactoff].count;
pg_read_barrier(); /* pairs with barrier in GetNewTransactionId() */
pgprocno = arrayP->pgprocnos[pgxactoff];
- proc = &allProcs[pgprocno];
+ proc = allProcs[pgprocno];
for (j = pxids - 1; j >= 0; j--)
{
/* Fetch xid just once - see GetNewTransactionId */
@@ -1622,7 +1622,6 @@ TransactionIdIsInProgress(TransactionId xid)
return false;
}
-
/*
* Determine XID horizons.
*
@@ -1740,7 +1739,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
for (int index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int8 statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
TransactionId xmin;
@@ -2224,7 +2223,7 @@ GetSnapshotData(Snapshot snapshot)
TransactionId xid = UINT32_ACCESS_ONCE(other_xids[pgxactoff]);
uint8 statusFlags;
- Assert(allProcs[arrayP->pgprocnos[pgxactoff]].pgxactoff == pgxactoff);
+ Assert(allProcs[arrayP->pgprocnos[pgxactoff]]->pgxactoff == pgxactoff);
/*
* If the transaction has no XID assigned, we can skip it; it
@@ -2298,7 +2297,7 @@ GetSnapshotData(Snapshot snapshot)
if (nsubxids > 0)
{
int pgprocno = pgprocnos[pgxactoff];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
pg_read_barrier(); /* pairs with GetNewTransactionId */
@@ -2499,7 +2498,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int statusFlags = ProcGlobal->statusFlags[index];
TransactionId xid;
@@ -2725,7 +2724,7 @@ GetRunningTransactionData(void)
if (TransactionIdPrecedes(xid, oldestDatabaseRunningXid))
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId == MyDatabaseId)
oldestDatabaseRunningXid = xid;
@@ -2756,7 +2755,7 @@ GetRunningTransactionData(void)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
int nsubxids;
/*
@@ -2858,7 +2857,7 @@ GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs)
{
TransactionId xid;
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(other_xids[index]);
@@ -3020,7 +3019,7 @@ GetVirtualXIDsDelayingChkpt(int *nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if ((proc->delayChkptFlags & type) != 0)
{
@@ -3061,7 +3060,7 @@ HaveVirtualXIDsDelayingChkpt(VirtualTransactionId *vxids, int nvxids, int type)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId vxid;
GET_VXID_FROM_PGPROC(vxid, *proc);
@@ -3189,7 +3188,7 @@ BackendPidGetProcWithLock(int pid)
for (index = 0; index < arrayP->numProcs; index++)
{
- PGPROC *proc = &allProcs[arrayP->pgprocnos[index]];
+ PGPROC *proc = allProcs[arrayP->pgprocnos[index]];
if (proc->pid == pid)
{
@@ -3232,7 +3231,7 @@ BackendXidGetPid(TransactionId xid)
if (other_xids[index] == xid)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
result = proc->pid;
break;
@@ -3301,7 +3300,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc == MyProc)
@@ -3403,7 +3402,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3468,7 +3467,7 @@ SignalVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode,
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
VirtualTransactionId procvxid;
GET_VXID_FROM_PGPROC(procvxid, *proc);
@@ -3523,7 +3522,7 @@ MinimumActiveBackends(int min)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
/*
* Since we're not holding a lock, need to be prepared to deal with
@@ -3569,7 +3568,7 @@ CountDBBackends(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3598,7 +3597,7 @@ CountDBConnections(Oid databaseid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3629,7 +3628,7 @@ CancelDBBackends(Oid databaseid, ProcSignalReason sigmode, bool conflictPending)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (databaseid == InvalidOid || proc->databaseId == databaseid)
{
@@ -3670,7 +3669,7 @@ CountUserBackends(Oid roleid)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->pid == 0)
continue; /* do not count prepared xacts */
@@ -3733,7 +3732,7 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < arrayP->numProcs; index++)
{
int pgprocno = arrayP->pgprocnos[index];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
uint8 statusFlags = ProcGlobal->statusFlags[index];
if (proc->databaseId != databaseId)
@@ -3799,7 +3798,7 @@ TerminateOtherDBBackends(Oid databaseId)
for (i = 0; i < procArray->numProcs; i++)
{
int pgprocno = arrayP->pgprocnos[i];
- PGPROC *proc = &allProcs[pgprocno];
+ PGPROC *proc = allProcs[pgprocno];
if (proc->databaseId != databaseId)
continue;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 62f3471448e..c84a2a5f1bc 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2844,7 +2844,7 @@ FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
@@ -3103,7 +3103,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
*/
for (i = 0; i < ProcGlobal->allProcCount; i++)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
uint32 j;
/* A backend never blocks itself */
@@ -3790,7 +3790,7 @@ GetLockStatusData(void)
*/
for (i = 0; i < ProcGlobal->allProcCount; ++i)
{
- PGPROC *proc = &ProcGlobal->allProcs[i];
+ PGPROC *proc = ProcGlobal->allProcs[i];
/* Skip backends with pid=0, as they don't hold fast-path locks */
if (proc->pid == 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..11259151a7d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -29,21 +29,29 @@
*/
#include "postgres.h"
+#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "access/transam.h"
#include "access/twophase.h"
#include "access/xlogutils.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_numa.h"
#include "postmaster/autovacuum.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
#include "storage/condition_variable.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -90,6 +98,31 @@ static void AuxiliaryProcKill(int code, Datum arg);
static void CheckDeadLock(void);
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_procs_per_node = -1; /* pgprocs per node */
+
+static Size get_memory_page_size(void); /* XXX duplicate with bufi_init.c */
+
+static void pgproc_partitions_prepare(void);
+static char *pgproc_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node);
+static char *fastpath_partition_init(char *ptr, int num_procs,
+ int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize);
+
+typedef struct PGProcPartition
+{
+ int num_procs;
+ int numa_node;
+ void *pgproc_ptr;
+ void *fastpath_ptr;
+} PGProcPartition;
+
+static PGProcPartition *partitions = NULL;
+
/*
* Report shared-memory space needed by PGPROC.
*/
@@ -100,11 +133,63 @@ PGProcShmemSize(void)
Size TotalProcs =
add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC *)));
size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+ /*
+ * To support NUMA partitioning, the PGPROC array will be divided into
+ * multiple chunks - one per NUMA node, and one extra for auxiliary/2PC
+ * entries (which are not assigned to any NUMA node).
+ *
+ * We can't simply map pages of a single continuous array, because the
+ * PGPROC entries are very small and too many of them would fit on a
+ * single page (at least with huge pages). Far more than reasonable values
+ * of max_connections. So instead we cut the array into separate pieces
+ * for each node.
+ *
+ * Each piece may need up to one memory page of padding, to make it
+ * aligned with memory page (for NUMA), So we just add a page - it's a bit
+ * wasteful, but should not matter much - NUMA is meant for large boxes,
+ * so a couple pages is negligible.
+ *
+ * We only do this with NUMA partitioning. With the GUC disabled, or when
+ * we find we can't do that for some reason, we just allocate the PGPROC
+ * array as a single chunk. This is determined by the earlier call to
+ * pgproc_partitions_prepare().
+ *
+ * XXX It might be more painful with very large huge pages (e.g. 1GB).
+ */
+
+ /*
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+
+ /*
+ * Also account for a small registry of partitions, a simple array of
+ * partitions at the beginning.
+ */
+ size = add_size(size, mul_size((numa_nodes + 1), sizeof(PGProcPartition)));
+ }
+ else
+ {
+ /* otherwise add only a tiny registry, with a single partition */
+ size = add_size(size, sizeof(PGProcPartition));
+ }
+
return size;
}
@@ -129,6 +214,25 @@ FastPathLockShmemSize(void)
size = add_size(size, mul_size(TotalProcs, (fpLockBitsSize + fpRelIdSize)));
+ /*
+ * When applying NUMA to the fast-path locks, we follow the same logic as
+ * for PGPROC entries. See the comments in PGProcShmemSize().
+ *
+ * If PGPROC partitioning is enabled, and we decided it's possible, we
+ * need to add one memory page per NUMA node (and one for auxiliary/2PC
+ * processes) to allow proper alignment.
+ *
+ * XXX This is a a bit wasteful, because it might actually add pages even
+ * when not strictly needed (if it's already aligned). And we always
+ * assume we'll add a whole page, even if the alignment needs only less
+ * memory.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ Assert(numa_nodes > 0);
+ size = add_size(size, mul_size((numa_nodes + 1), numa_page_size));
+ }
+
return size;
}
@@ -140,6 +244,9 @@ ProcGlobalShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for pgproc entries etc */
+ pgproc_partitions_prepare();
+
/* ProcGlobal */
size = add_size(size, sizeof(PROC_HDR));
size = add_size(size, sizeof(slock_t));
@@ -191,7 +298,7 @@ ProcGlobalSemas(void)
void
InitProcGlobal(void)
{
- PGPROC *procs;
+ PGPROC **procs;
int i,
j;
bool found;
@@ -205,6 +312,8 @@ InitProcGlobal(void)
Size requestSize;
char *ptr;
+ Size mem_page_size = get_memory_page_size();
+
/* Create the ProcGlobal shared structure */
ProcGlobal = (PROC_HDR *)
ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
@@ -241,19 +350,115 @@ InitProcGlobal(void)
MemSet(ptr, 0, requestSize);
- procs = (PGPROC *) ptr;
- ptr = (char *) ptr + TotalProcs * sizeof(PGPROC);
+ /* allprocs (array of pointers to PGPROC entries) */
+ procs = (PGPROC **) ptr;
+ ptr = (char *) ptr + TotalProcs * sizeof(PGPROC *);
ProcGlobal->allProcs = procs;
/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
+ {
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+ Assert(numa_nodes > 0);
+
+ /*
+ * Now initialize the PGPROC partition registry with one partitoion
+ * per NUMA node.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += (numa_nodes * sizeof(PGProcPartition));
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[i].num_procs = node_procs;
+ partitions[i].numa_node = i;
+ partitions[i].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, i);
+
+ total_procs += node_procs;
+
+ /* don't underflow/overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
+ Assert(total_procs == MaxBackends);
+
+ /*
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
+ */
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
+
+ /* make sure to align the PGPROC array to memory page */
+ ptr = (char *) TYPEALIGN(numa_page_size, ptr);
+
+ /* fill in the partition info */
+ partitions[numa_nodes].num_procs = node_procs;
+ partitions[numa_nodes].numa_node = -1;
+ partitions[numa_nodes].pgproc_ptr = ptr;
+
+ ptr = pgproc_partition_init(ptr, node_procs, total_procs, -1);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /*
+ * Now initialize the PGPROC partition registry with a single
+ * partition for all the procs.
+ */
+ partitions = (PGProcPartition *) ptr;
+ ptr += sizeof(PGProcPartition);
+
+ /* just treat everything as a single array, with no alignment */
+ ptr = pgproc_partition_init(ptr, TotalProcs, 0, -1);
+
+ /* fill in the partition info */
+ partitions[0].num_procs = TotalProcs;
+ partitions[0].numa_node = -1;
+ partitions[0].pgproc_ptr = ptr;
+
+ /* don't overflow the allocation */
+ Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
+ }
+
/*
* Allocate arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
* how hotly they are accessed.
+ *
+ * XXX Would it make sense to NUMA-partition these chunks too, somehow?
+ * But those arrays are tiny, fit into a single memory page, so would need
+ * to be made more complex. Not sure.
*/
ProcGlobal->xids = (TransactionId *) ptr;
ptr = (char *) ptr + (TotalProcs * sizeof(*ProcGlobal->xids));
@@ -286,24 +491,92 @@ InitProcGlobal(void)
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- for (i = 0; i < TotalProcs; i++)
+ /*
+ * Mimic the logic we used to partition PGPROC entries.
+ */
+
+ /*
+ * If NUMA partitioning is enabled, and we decided we actually can do the
+ * partitioning, allocate the chunks.
+ *
+ * Otherwise we'll allocate a single array for everything. It's not quite
+ * what we did without NUMA, because there's an extra level of
+ * indirection, but it's the best we can do.
+ */
+ if (numa_procs_interleave && numa_can_partition)
{
- PGPROC *proc = &procs[i];
+ int node_procs;
+ int total_procs = 0;
+
+ Assert(numa_procs_per_node > 0);
+
+ /* build PGPROC entries for NUMA nodes */
+ for (i = 0; i < numa_nodes; i++)
+ {
+ /* the last NUMA node may get fewer PGPROC entries, but meh */
+ node_procs = Min(numa_procs_per_node, MaxBackends - total_procs);
+
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(mem_page_size, fpPtr);
- /* Common initialization for all PGPROCs, regardless of type. */
+ /* remember this pointer too */
+ partitions[i].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[i].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, i,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ Assert(total_procs == MaxBackends);
/*
- * Set the fast-path lock arrays, and move the pointer. We interleave
- * the two arrays, to (hopefully) get some locality for each backend.
+ * Also build PGPROC entries for auxiliary procs / prepared xacts (we
+ * however don't assign those to any NUMA node).
*/
- proc->fpLockBits = (uint64 *) fpPtr;
- fpPtr += fpLockBitsSize;
+ node_procs = (NUM_AUXILIARY_PROCS + max_prepared_xacts);
- proc->fpRelId = (Oid *) fpPtr;
- fpPtr += fpRelIdSize;
+ /* make sure to align the PGPROC array to memory page */
+ fpPtr = (char *) TYPEALIGN(numa_page_size, fpPtr);
+ /* remember this pointer too */
+ partitions[numa_nodes].fastpath_ptr = fpPtr;
+ Assert(node_procs == partitions[numa_nodes].num_procs);
+
+ fpPtr = fastpath_partition_init(fpPtr, node_procs, total_procs, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ total_procs += node_procs;
+
+ /* don't overflow the allocation */
Assert(fpPtr <= fpEndPtr);
+ Assert(total_procs = TotalProcs);
+ }
+ else
+ {
+ /* remember this pointer too */
+ partitions[0].fastpath_ptr = fpPtr;
+ Assert(TotalProcs == partitions[0].num_procs);
+
+ /* just treat everything as a single array, with no alignment */
+ fpPtr = fastpath_partition_init(fpPtr, TotalProcs, 0, -1,
+ fpLockBitsSize, fpRelIdSize);
+
+ /* don't overflow the allocation */
+ Assert(fpPtr <= fpEndPtr);
+ }
+
+ for (i = 0; i < TotalProcs; i++)
+ {
+ PGPROC *proc = procs[i];
+
+ Assert(proc->procnumber == i);
+
/*
* Set up per-PGPROC semaphore, latch, and fpInfoLock. Prepared xact
* dummy PGPROCs don't need these though - they're never associated
@@ -366,15 +639,12 @@ InitProcGlobal(void)
pg_atomic_init_u64(&(proc->waitStart), 0);
}
- /* Should have consumed exactly the expected amount of fast-path memory. */
- Assert(fpPtr == fpEndPtr);
-
/*
* Save pointers to the blocks of PGPROC structures reserved for auxiliary
* processes and prepared transactions.
*/
- AuxiliaryProcs = &procs[MaxBackends];
- PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+ AuxiliaryProcs = procs[MaxBackends];
+ PreparedXactProcs = procs[MaxBackends + NUM_AUXILIARY_PROCS];
/* Create ProcStructLock spinlock, too */
ProcStructLock = (slock_t *) ShmemInitStruct("ProcStructLock spinlock",
@@ -435,7 +705,45 @@ InitProcess(void)
if (!dlist_is_empty(procgloballist))
{
- MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+ /*
+ * With numa interleaving of PGPROC, try to get a PROC entry from the
+ * right NUMA node (when the process starts).
+ *
+ * XXX The process may move to a different NUMA node later, but
+ * there's not much we can do about that.
+ */
+ if (numa_procs_interleave)
+ {
+ dlist_mutable_iter iter;
+ unsigned cpu;
+ unsigned node;
+ int rc;
+
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ MyProc = NULL;
+
+ dlist_foreach_modify(iter, procgloballist)
+ {
+ PGPROC *proc;
+
+ proc = dlist_container(PGPROC, links, iter.cur);
+
+ if (proc->numa_node == node)
+ {
+ MyProc = proc;
+ dlist_delete(iter.cur);
+ break;
+ }
+ }
+ }
+
+ /* didn't find PGPROC from the correct NUMA node, pick any free one */
+ if (MyProc == NULL)
+ MyProc = dlist_container(PGPROC, links, dlist_pop_head_node(procgloballist));
+
SpinLockRelease(ProcStructLock);
}
else
@@ -1988,7 +2296,7 @@ ProcSendSignal(ProcNumber procNumber)
if (procNumber < 0 || procNumber >= ProcGlobal->allProcCount)
elog(ERROR, "procNumber out of range");
- SetLatch(&ProcGlobal->allProcs[procNumber].procLatch);
+ SetLatch(&ProcGlobal->allProcs[procNumber]->procLatch);
}
/*
@@ -2063,3 +2371,222 @@ BecomeLockGroupMember(PGPROC *leader, int pid)
return ok;
}
+
+/* copy from buf_init.c */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /*
+ * XXX This is a bit annoying/confusing, because we may get a different
+ * result depending on when we call it. Before mmap() we don't know if the
+ * huge pages get used, so we assume they will. And then if we don't get
+ * huge pages, we'll waste memory etc.
+ */
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status == HUGE_PAGES_OFF)
+ huge_page_size = 0;
+ else
+ GetHugePageSize(&huge_page_size, NULL);
+
+ return Max(os_page_size, huge_page_size);
+}
+
+/*
+ * pgproc_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * NUMA partitioning
+ *
+ * Now build the actual PGPROC arrays, one "chunk" per NUMA node (and one
+ * extra for auxiliary processes and 2PC transactions, not associated with
+ * any particular node).
+ *
+ * First determine how many "backend" procs to allocate per NUMA node. The
+ * count may not be exactly divisible, but we mostly ignore that. The last
+ * node may get somewhat fewer PGPROC entries, but the imbalance ought to
+ * be pretty small (if MaxBackends >> numa_nodes).
+ *
+ * XXX A fairer distribution is possible, but not worth it for now.
+ */
+static void
+pgproc_partitions_prepare(void)
+{
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ numa_procs_per_node = (MaxBackends + (numa_nodes - 1)) / numa_nodes;
+
+ elog(LOG, "NUMA: pgproc backends %d num_nodes %d per_node %d",
+ MaxBackends, numa_nodes, numa_procs_per_node);
+
+ Assert(numa_nodes * numa_procs_per_node >= MaxBackends);
+
+ /* success */
+ numa_can_partition = true;
+}
+
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+/*
+ * doesn't do alignment
+ */
+static char *
+pgproc_partition_init(char *ptr, int num_procs, int allprocs_index, int node)
+{
+ PGPROC *procs_node;
+
+ /* allocate the PGPROC chunk for this node */
+ procs_node = (PGPROC *) ptr;
+
+ /* pointer right after this array */
+ ptr = (char *) ptr + num_procs * sizeof(PGPROC);
+
+ elog(LOG, "NUMA: pgproc_init_partition procs %p endptr %p num_procs %d node %d",
+ procs_node, ptr, num_procs, node);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node((char *) procs_node, ptr, node);
+
+ /* add pointers to the PGPROC entries to allProcs */
+ for (int i = 0; i < num_procs; i++)
+ {
+ procs_node[i].numa_node = node;
+ procs_node[i].procnumber = allprocs_index;
+
+ ProcGlobal->allProcs[allprocs_index] = &procs_node[i];
+
+ allprocs_index++;
+ }
+
+ return ptr;
+}
+
+static char *
+fastpath_partition_init(char *ptr, int num_procs, int allprocs_index, int node,
+ Size fpLockBitsSize, Size fpRelIdSize)
+{
+ char *endptr = ptr + num_procs * (fpLockBitsSize + fpRelIdSize);
+
+ /*
+ * if node specified, move to node - do this before we start touching the
+ * memory, to make sure it's not mapped to any node yet
+ */
+ if (node != -1)
+ pg_numa_move_to_node(ptr, endptr, node);
+
+ /*
+ * Now point the PGPROC entries to the fast-path arrays, and also advance
+ * the fpPtr.
+ */
+ for (int i = 0; i < num_procs; i++)
+ {
+ PGPROC *proc = ProcGlobal->allProcs[allprocs_index];
+
+ /* cross-check we got the expected NUMA node */
+ Assert(proc->numa_node == node);
+ Assert(proc->procnumber == allprocs_index);
+
+ /*
+ * Set the fast-path lock arrays, and move the pointer. We interleave
+ * the two arrays, to (hopefully) get some locality for each backend.
+ */
+ proc->fpLockBits = (uint64 *) ptr;
+ ptr += fpLockBitsSize;
+
+ proc->fpRelId = (Oid *) ptr;
+ ptr += fpRelIdSize;
+
+ Assert(ptr <= endptr);
+
+ allprocs_index++;
+ }
+
+ Assert(ptr == endptr);
+
+ return endptr;
+}
+
+int
+ProcPartitionCount(void)
+{
+ if (numa_procs_interleave && numa_can_partition)
+ return (numa_nodes + 1);
+
+ return 1;
+}
+
+void
+ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr)
+{
+ PGProcPartition *part = &partitions[idx];
+
+ Assert((idx >= 0) && (idx < ProcPartitionCount()));
+
+ *nprocs = part->num_procs;
+ *procsptr = part->pgproc_ptr;
+ *fpptr = part->fastpath_ptr;
+ *node = part->numa_node;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a11bc71a386..6ee4684d1b8 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -149,6 +149,7 @@ int MaxBackends = 0;
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
bool numa_partition_freelist = false;
+bool numa_procs_interleave = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0552ed62cc7..7b718760248 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2146,6 +2146,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_procs_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of PGPROC entries."),
+ gettext_noop("When enabled, the PGPROC entries are interleaved to all NUMA nodes."),
+ },
+ &numa_procs_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 66baf2bf33e..cdeee8dccba 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -181,6 +181,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT bool numa_partition_freelist;
+extern PGDLLIMPORT bool numa_procs_interleave;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index c6f5ebceefd..d2d269941fc 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -202,6 +202,8 @@ struct PGPROC
* vacuum must not remove tuples deleted by
* xid >= xmin ! */
+ int procnumber; /* index in ProcGlobal->allProcs */
+
int pid; /* Backend's process ID; 0 if prepared xact */
int pgxactoff; /* offset into various ProcGlobal->arrays with
@@ -327,6 +329,9 @@ struct PGPROC
PGPROC *lockGroupLeader; /* lock group leader, if I'm a member */
dlist_head lockGroupMembers; /* list of members, if I'm a leader */
dlist_node lockGroupLink; /* my member link, if I'm a member */
+
+ /* NUMA node */
+ int numa_node;
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. */
@@ -391,7 +396,7 @@ extern PGDLLIMPORT PGPROC *MyProc;
typedef struct PROC_HDR
{
/* Array of PGPROC structures (not including dummies for prepared txns) */
- PGPROC *allProcs;
+ PGPROC **allProcs;
/* Array mirroring PGPROC.xid for each PGPROC currently in the procarray */
TransactionId *xids;
@@ -443,8 +448,8 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
/*
* Accessors for getting PGPROC given a ProcNumber and vice versa.
*/
-#define GetPGProcByNumber(n) (&ProcGlobal->allProcs[(n)])
-#define GetNumberFromPGProc(proc) ((proc) - &ProcGlobal->allProcs[0])
+#define GetPGProcByNumber(n) (ProcGlobal->allProcs[(n)])
+#define GetNumberFromPGProc(proc) ((proc)->procnumber)
/*
* We set aside some extra PGPROC structures for "special worker" processes,
@@ -520,4 +525,7 @@ extern PGPROC *AuxiliaryPidGetProc(int pid);
extern void BecomeLockGroupLeader(void);
extern bool BecomeLockGroupMember(PGPROC *leader, int pid);
+extern int ProcPartitionCount(void);
+extern void ProcPartitionGet(int idx, int *node, int *nprocs, void **procsptr, void **fpptr);
+
#endif /* _PROC_H_ */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a38dd8d6242..5595cd48eee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1877,6 +1877,7 @@ PGP_MPI
PGP_PubKey
PGP_S2K
PGPing
+PGProcPartition
PGQueryClass
PGRUsage
PGSemaphore
--
2.50.1
[text/x-patch] v3-0005-NUMA-clockweep-partitioning.patch (39.4K, ../[email protected]/4-v3-0005-NUMA-clockweep-partitioning.patch)
download | inline diff:
From 2bfd4a824b12e9a865c5ef0a8ed33e215fb1b698 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Sun, 8 Jun 2025 18:53:12 +0200
Subject: [PATCH v3 5/7] NUMA: clockweep partitioning
Similar to the frelist patch - partition the "clocksweep" algorithm to
work on the sequence of smaller partitions, one by one.
It extends the "pg_buffercache_partitions" view to include information
about the clocksweep activity.
Note: This needs some sort of "balancing" when one of the partitions is
much busier than the rest (e.g. because there's a single backend consuming
a lot of buffers from it).
---
.../pg_buffercache--1.6--1.7.sql | 5 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/bufmgr.c | 476 ++++++++++--------
src/backend/storage/buffer/freelist.c | 224 +++++++--
src/include/storage/buf_internals.h | 4 +-
src/include/storage/bufmgr.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 478 insertions(+), 261 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index 3871c261528..b7d8ea45ed7 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -12,7 +12,10 @@ LANGUAGE C PARALLEL SAFE;
-- Create a view for convenient access.
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
- (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer, buffers_consumed bigint, buffers_remain bigint, buffers_free bigint);
+ (partition integer,
+ numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer,
+ buffers_consumed bigint, buffers_remain bigint, buffers_free bigint,
+ complete_passes bigint, buffer_allocs bigint, next_victim_buffer integer);
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 668ada8c47b..5169655ae78 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 11
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -818,6 +818,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 8, "buffers_free",
INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 9, "complete_passes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 10, "buffer_allocs",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 11, "next_victim_buffer",
+ INT4OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -843,6 +849,10 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
buffers_remain,
buffers_free;
+ uint32 complete_passes,
+ buffer_allocs,
+ next_victim_buffer;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
@@ -850,7 +860,8 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
&first_buffer, &last_buffer);
FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
- &buffers_free);
+ &buffers_free, &complete_passes,
+ &buffer_allocs, &next_victim_buffer);
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -876,6 +887,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[7] = Int64GetDatum(buffers_free);
nulls[7] = false;
+ values[8] = Int32GetDatum(complete_passes);
+ nulls[8] = false;
+
+ values[9] = Int32GetDatum(buffer_allocs);
+ nulls[9] = false;
+
+ values[10] = Int32GetDatum(next_victim_buffer);
+ nulls[10] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5922689fe5d..bd007c1c621 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3587,6 +3587,23 @@ BufferSync(int flags)
TRACE_POSTGRESQL_BUFFER_SYNC_DONE(NBuffers, num_written, num_to_scan);
}
+/*
+ * Information saved between calls so we can determine the strategy
+ * point's advance rate and avoid scanning already-cleaned buffers.
+ *
+ * XXX One value per partition. We don't know how many partitions are
+ * there, so allocate 32, should be enough for the PoC patch.
+ *
+ * XXX might be better to have a per-partition struct with all the info
+ */
+#define MAX_CLOCKSWEEP_PARTITIONS 32
+static bool saved_info_valid = false;
+static int prev_strategy_buf_id[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 prev_strategy_passes[MAX_CLOCKSWEEP_PARTITIONS];
+static int next_to_clean[MAX_CLOCKSWEEP_PARTITIONS];
+static uint32 next_passes[MAX_CLOCKSWEEP_PARTITIONS];
+
+
/*
* BgBufferSync -- Write out some dirty buffers in the pool.
*
@@ -3602,55 +3619,24 @@ bool
BgBufferSync(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
- int strategy_buf_id;
- uint32 strategy_passes;
uint32 recent_alloc;
+ uint32 recent_alloc_partition;
+ int num_partitions;
- /*
- * Information saved between calls so we can determine the strategy
- * point's advance rate and avoid scanning already-cleaned buffers.
- */
- static bool saved_info_valid = false;
- static int prev_strategy_buf_id;
- static uint32 prev_strategy_passes;
- static int next_to_clean;
- static uint32 next_passes;
-
- /* Moving averages of allocation rate and clean-buffer density */
- static float smoothed_alloc = 0;
- static float smoothed_density = 10.0;
-
- /* Potentially these could be tunables, but for now, not */
- float smoothing_samples = 16;
- float scan_whole_pool_milliseconds = 120000.0;
-
- /* Used to compute how far we scan ahead */
- long strategy_delta;
- int bufs_to_lap;
- int bufs_ahead;
- float scans_per_alloc;
- int reusable_buffers_est;
- int upcoming_alloc_est;
- int min_scan_buffers;
-
- /* Variables for the scanning loop proper */
- int num_to_scan;
- int num_written;
- int reusable_buffers;
+ /* assume we can hibernate, any partition can set to false */
+ bool hibernate = true;
- /* Variables for final smoothed_density update */
- long new_strategy_delta;
- uint32 new_recent_alloc;
+ /* get the number of clocksweep partitions, and total alloc count */
+ StrategySyncPrepare(&num_partitions, &recent_alloc);
- /*
- * Find out where the freelist clock sweep currently is, and how many
- * buffer allocations have happened since our last call.
- */
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ Assert(num_partitions <= MAX_CLOCKSWEEP_PARTITIONS);
/* Report buffer alloc counts to pgstat */
PendingBgWriterStats.buf_alloc += recent_alloc;
+ /* average alloc buffers per partition */
+ recent_alloc_partition = (recent_alloc / num_partitions);
+
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -3663,223 +3649,285 @@ BgBufferSync(WritebackContext *wb_context)
}
/*
- * Compute strategy_delta = how many buffers have been scanned by the
- * clock sweep since last time. If first time through, assume none. Then
- * see if we are still ahead of the clock sweep, and if so, how many
- * buffers we could scan before we'd catch up with it and "lap" it. Note:
- * weird-looking coding of xxx_passes comparisons are to avoid bogus
- * behavior when the passes counts wrap around.
- */
- if (saved_info_valid)
- {
- int32 passes_delta = strategy_passes - prev_strategy_passes;
-
- strategy_delta = strategy_buf_id - prev_strategy_buf_id;
- strategy_delta += (long) passes_delta * NBuffers;
+ * now process the clocksweep partitions, one by one, using the same
+ * cleanup that we used for all buffers
+ *
+ * XXX Maybe we should randomize the order of partitions a bit, so that we
+ * don't start from partition 0 all the time? Perhaps not entirely, but at
+ * least pick a random starting point?
+ */
+ for (int partition = 0; partition < num_partitions; partition++)
+ {
+ /* info obtained from freelist.c */
+ int strategy_buf_id;
+ uint32 strategy_passes;
+
+ /* Moving averages of allocation rate and clean-buffer density */
+ static float smoothed_alloc = 0;
+ static float smoothed_density = 10.0;
+
+ /* Potentially these could be tunables, but for now, not */
+ float smoothing_samples = 16;
+ float scan_whole_pool_milliseconds = 120000.0;
+
+ /* Used to compute how far we scan ahead */
+ long strategy_delta;
+ int bufs_to_lap;
+ int bufs_ahead;
+ float scans_per_alloc;
+ int reusable_buffers_est;
+ int upcoming_alloc_est;
+ int min_scan_buffers;
+
+ /* Variables for the scanning loop proper */
+ int num_to_scan;
+ int num_written;
+ int reusable_buffers;
+
+ /* Variables for final smoothed_density update */
+ long new_strategy_delta;
+ uint32 new_recent_alloc;
+
+ /* buffer range for the clocksweep partition */
+ int first_buffer;
+ int num_buffers;
- Assert(strategy_delta >= 0);
+ /*
+ * Find out where the freelist clock sweep currently is, and how many
+ * buffer allocations have happened since our last call.
+ */
+ strategy_buf_id = StrategySyncStart(partition, &strategy_passes,
+ &first_buffer, &num_buffers);
- if ((int32) (next_passes - strategy_passes) > 0)
+ /*
+ * Compute strategy_delta = how many buffers have been scanned by the
+ * clock sweep since last time. If first time through, assume none.
+ * Then see if we are still ahead of the clock sweep, and if so, how
+ * many buffers we could scan before we'd catch up with it and "lap"
+ * it. Note: weird-looking coding of xxx_passes comparisons are to
+ * avoid bogus behavior when the passes counts wrap around.
+ */
+ if (saved_info_valid)
{
- /* we're one pass ahead of the strategy point */
- bufs_to_lap = strategy_buf_id - next_to_clean;
+ int32 passes_delta = strategy_passes - prev_strategy_passes[partition];
+
+ strategy_delta = strategy_buf_id - prev_strategy_buf_id[partition];
+ strategy_delta += (long) passes_delta * num_buffers;
+
+ Assert(strategy_delta >= 0);
+
+ if ((int32) (next_passes[partition] - strategy_passes) > 0)
+ {
+ /* we're one pass ahead of the strategy point */
+ bufs_to_lap = strategy_buf_id - next_to_clean[partition];
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
#endif
- }
- else if (next_passes == strategy_passes &&
- next_to_clean >= strategy_buf_id)
- {
- /* on same pass, but ahead or at least not behind */
- bufs_to_lap = NBuffers - (next_to_clean - strategy_buf_id);
+ }
+ else if (next_passes[partition] == strategy_passes &&
+ next_to_clean[partition] >= strategy_buf_id)
+ {
+ /* on same pass, but ahead or at least not behind */
+ bufs_to_lap = num_buffers - (next_to_clean[partition] - strategy_buf_id);
+#ifdef BGW_DEBUG
+ elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta, bufs_to_lap);
+#endif
+ }
+ else
+ {
+ /*
+ * We're behind, so skip forward to the strategy point and
+ * start cleaning from there.
+ */
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter ahead: bgw %u-%u strategy %u-%u delta=%ld lap=%d",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta, bufs_to_lap);
+ elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
+ next_passes, next_to_clean,
+ strategy_passes, strategy_buf_id,
+ strategy_delta);
#endif
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
+ }
}
else
{
/*
- * We're behind, so skip forward to the strategy point and start
- * cleaning from there.
+ * Initializing at startup or after LRU scanning had been off.
+ * Always start at the strategy point.
*/
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter behind: bgw %u-%u strategy %u-%u delta=%ld",
- next_passes, next_to_clean,
- strategy_passes, strategy_buf_id,
- strategy_delta);
+ elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
+ strategy_passes, strategy_buf_id);
#endif
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
+ strategy_delta = 0;
+ next_to_clean[partition] = strategy_buf_id;
+ next_passes[partition] = strategy_passes;
+ bufs_to_lap = num_buffers;
}
- }
- else
- {
- /*
- * Initializing at startup or after LRU scanning had been off. Always
- * start at the strategy point.
- */
-#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter initializing: strategy %u-%u",
- strategy_passes, strategy_buf_id);
-#endif
- strategy_delta = 0;
- next_to_clean = strategy_buf_id;
- next_passes = strategy_passes;
- bufs_to_lap = NBuffers;
- }
- /* Update saved info for next time */
- prev_strategy_buf_id = strategy_buf_id;
- prev_strategy_passes = strategy_passes;
- saved_info_valid = true;
+ /* Update saved info for next time */
+ prev_strategy_buf_id[partition] = strategy_buf_id;
+ prev_strategy_passes[partition] = strategy_passes;
+ /* FIXME has to happen after all partitions */
+ /* saved_info_valid = true; */
- /*
- * Compute how many buffers had to be scanned for each new allocation, ie,
- * 1/density of reusable buffers, and track a moving average of that.
- *
- * If the strategy point didn't move, we don't update the density estimate
- */
- if (strategy_delta > 0 && recent_alloc > 0)
- {
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
- }
+ /*
+ * Compute how many buffers had to be scanned for each new allocation,
+ * ie, 1/density of reusable buffers, and track a moving average of
+ * that.
+ *
+ * If the strategy point didn't move, we don't update the density
+ * estimate
+ */
+ if (strategy_delta > 0 && recent_alloc_partition > 0)
+ {
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_partition;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
+ }
- /*
- * Estimate how many reusable buffers there are between the current
- * strategy point and where we've scanned ahead to, based on the smoothed
- * density estimate.
- */
- bufs_ahead = NBuffers - bufs_to_lap;
- reusable_buffers_est = (float) bufs_ahead / smoothed_density;
+ /*
+ * Estimate how many reusable buffers there are between the current
+ * strategy point and where we've scanned ahead to, based on the
+ * smoothed density estimate.
+ */
+ bufs_ahead = num_buffers - bufs_to_lap;
+ reusable_buffers_est = (float) bufs_ahead / smoothed_density;
- /*
- * Track a moving average of recent buffer allocations. Here, rather than
- * a true average we want a fast-attack, slow-decline behavior: we
- * immediately follow any increase.
- */
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
- else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
- smoothing_samples;
+ /*
+ * Track a moving average of recent buffer allocations. Here, rather
+ * than a true average we want a fast-attack, slow-decline behavior:
+ * we immediately follow any increase.
+ */
+ if (smoothed_alloc <= (float) recent_alloc_partition)
+ smoothed_alloc = recent_alloc_partition;
+ else
+ smoothed_alloc += ((float) recent_alloc_partition - smoothed_alloc) /
+ smoothing_samples;
- /* Scale the estimate by a GUC to allow more aggressive tuning. */
- upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
+ /* Scale the estimate by a GUC to allow more aggressive tuning. */
+ upcoming_alloc_est = (int) (smoothed_alloc * bgwriter_lru_multiplier);
- /*
- * If recent_alloc remains at zero for many cycles, smoothed_alloc will
- * eventually underflow to zero, and the underflows produce annoying
- * kernel warnings on some platforms. Once upcoming_alloc_est has gone to
- * zero, there's no point in tracking smaller and smaller values of
- * smoothed_alloc, so just reset it to exactly zero to avoid this
- * syndrome. It will pop back up as soon as recent_alloc increases.
- */
- if (upcoming_alloc_est == 0)
- smoothed_alloc = 0;
+ /*
+ * If recent_alloc remains at zero for many cycles, smoothed_alloc
+ * will eventually underflow to zero, and the underflows produce
+ * annoying kernel warnings on some platforms. Once
+ * upcoming_alloc_est has gone to zero, there's no point in tracking
+ * smaller and smaller values of smoothed_alloc, so just reset it to
+ * exactly zero to avoid this syndrome. It will pop back up as soon
+ * as recent_alloc increases.
+ */
+ if (upcoming_alloc_est == 0)
+ smoothed_alloc = 0;
- /*
- * Even in cases where there's been little or no buffer allocation
- * activity, we want to make a small amount of progress through the buffer
- * cache so that as many reusable buffers as possible are clean after an
- * idle period.
- *
- * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many times
- * the BGW will be called during the scan_whole_pool time; slice the
- * buffer pool into that many sections.
- */
- min_scan_buffers = (int) (NBuffers / (scan_whole_pool_milliseconds / BgWriterDelay));
+ /*
+ * Even in cases where there's been little or no buffer allocation
+ * activity, we want to make a small amount of progress through the
+ * buffer cache so that as many reusable buffers as possible are clean
+ * after an idle period.
+ *
+ * (scan_whole_pool_milliseconds / BgWriterDelay) computes how many
+ * times the BGW will be called during the scan_whole_pool time; slice
+ * the buffer pool into that many sections.
+ */
+ min_scan_buffers = (int) (num_buffers / (scan_whole_pool_milliseconds / BgWriterDelay));
- if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
- {
+ if (upcoming_alloc_est < (min_scan_buffers + reusable_buffers_est))
+ {
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
- upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
+ elog(DEBUG2, "bgwriter: alloc_est=%d too small, using min=%d + reusable_est=%d",
+ upcoming_alloc_est, min_scan_buffers, reusable_buffers_est);
#endif
- upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
- }
-
- /*
- * Now write out dirty reusable buffers, working forward from the
- * next_to_clean point, until we have lapped the strategy scan, or cleaned
- * enough buffers to match our estimate of the next cycle's allocation
- * requirements, or hit the bgwriter_lru_maxpages limit.
- */
+ upcoming_alloc_est = min_scan_buffers + reusable_buffers_est;
+ }
- num_to_scan = bufs_to_lap;
- num_written = 0;
- reusable_buffers = reusable_buffers_est;
+ /*
+ * Now write out dirty reusable buffers, working forward from the
+ * next_to_clean point, until we have lapped the strategy scan, or
+ * cleaned enough buffers to match our estimate of the next cycle's
+ * allocation requirements, or hit the bgwriter_lru_maxpages limit.
+ */
- /* Execute the LRU scan */
- while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
- {
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ num_to_scan = bufs_to_lap;
+ num_written = 0;
+ reusable_buffers = reusable_buffers_est;
- if (++next_to_clean >= NBuffers)
+ /* Execute the LRU scan */
+ while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- next_to_clean = 0;
- next_passes++;
- }
- num_to_scan--;
+ int sync_state = SyncOneBuffer(next_to_clean[partition], true,
+ wb_context);
- if (sync_state & BUF_WRITTEN)
- {
- reusable_buffers++;
- if (++num_written >= bgwriter_lru_maxpages)
+ if (++next_to_clean[partition] >= (first_buffer + num_buffers))
{
- PendingBgWriterStats.maxwritten_clean++;
- break;
+ next_to_clean[partition] = first_buffer;
+ next_passes[partition]++;
+ }
+ num_to_scan--;
+
+ if (sync_state & BUF_WRITTEN)
+ {
+ reusable_buffers++;
+ if (++num_written >= (bgwriter_lru_maxpages / num_partitions))
+ {
+ PendingBgWriterStats.maxwritten_clean++;
+ break;
+ }
}
+ else if (sync_state & BUF_REUSABLE)
+ reusable_buffers++;
}
- else if (sync_state & BUF_REUSABLE)
- reusable_buffers++;
- }
- PendingBgWriterStats.buf_written_clean += num_written;
+ PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
- elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
- recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
- smoothed_density, reusable_buffers_est, upcoming_alloc_est,
- bufs_to_lap - num_to_scan,
- num_written,
- reusable_buffers - reusable_buffers_est);
+ elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
+ recent_alloc_partition, smoothed_alloc, strategy_delta, bufs_ahead,
+ smoothed_density, reusable_buffers_est, upcoming_alloc_est,
+ bufs_to_lap - num_to_scan,
+ num_written,
+ reusable_buffers - reusable_buffers_est);
#endif
- /*
- * Consider the above scan as being like a new allocation scan.
- * Characterize its density and update the smoothed one based on it. This
- * effectively halves the moving average period in cases where both the
- * strategy and the background writer are doing some useful scanning,
- * which is helpful because a long memory isn't as desirable on the
- * density estimates.
- */
- new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
- {
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
- smoothed_density += (scans_per_alloc - smoothed_density) /
- smoothing_samples;
+ /*
+ * Consider the above scan as being like a new allocation scan.
+ * Characterize its density and update the smoothed one based on it.
+ * This effectively halves the moving average period in cases where
+ * both the strategy and the background writer are doing some useful
+ * scanning, which is helpful because a long memory isn't as desirable
+ * on the density estimates.
+ */
+ new_strategy_delta = bufs_to_lap - num_to_scan;
+ new_recent_alloc = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ {
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ smoothed_density += (scans_per_alloc - smoothed_density) /
+ smoothing_samples;
#ifdef BGW_DEBUG
- elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
- scans_per_alloc, smoothed_density);
+ elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
+ new_recent_alloc, new_strategy_delta,
+ scans_per_alloc, smoothed_density);
#endif
+ }
+
+ /* hibernate if all partitions can hibernate */
+ hibernate &= (bufs_to_lap == 0 && recent_alloc_partition == 0);
}
+ /* now that we've scanned all partitions, mark the cached info as valid */
+ saved_info_valid = true;
+
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return hibernate;
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index c3fbd651dd5..ff02dc8e00b 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -63,17 +63,27 @@ typedef struct BufferStrategyFreelist
#define MIN_FREELIST_PARTITIONS 4
/*
- * The shared freelist control information.
+ * Information about one partition of the ClockSweep (on a subset of buffers).
+ *
+ * XXX Should be careful to align this to cachelines, etc.
*/
typedef struct
{
/* Spinlock: protects the values below */
- slock_t buffer_strategy_lock;
+ slock_t clock_sweep_lock;
+
+ /* range for this clock weep partition */
+ int32 firstBuffer;
+ int32 numBuffers;
/*
* Clock sweep hand: index of next buffer to consider grabbing. Note that
* this isn't a concrete buffer - we only ever increase the value. So, to
* get an actual buffer, it needs to be used modulo NBuffers.
+ *
+ * XXX This is relative to firstBuffer, so needs to be offset properly.
+ *
+ * XXX firstBuffer + (nextVictimBuffer % numBuffers)
*/
pg_atomic_uint32 nextVictimBuffer;
@@ -83,6 +93,15 @@ typedef struct
*/
uint32 completePasses; /* Complete cycles of the clock sweep */
pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+} ClockSweep;
+
+/*
+ * The shared freelist control information.
+ */
+typedef struct
+{
+ /* Spinlock: protects the values below */
+ slock_t buffer_strategy_lock;
/*
* Bgworker process to be notified upon activity or -1 if none. See
@@ -99,6 +118,9 @@ typedef struct
int num_partitions;
int num_partitions_per_node;
+ /* clocksweep partitions */
+ ClockSweep *sweeps;
+
BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
@@ -138,6 +160,7 @@ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy,
uint32 *buf_state);
static void AddBufferToRing(BufferAccessStrategy strategy,
BufferDesc *buf);
+static ClockSweep *ChooseClockSweep(void);
/*
* ClockSweepTick - Helper routine for StrategyGetBuffer()
@@ -149,6 +172,7 @@ static inline uint32
ClockSweepTick(void)
{
uint32 victim;
+ ClockSweep *sweep = ChooseClockSweep();
/*
* Atomically move hand ahead one buffer - if there's several processes
@@ -156,14 +180,14 @@ ClockSweepTick(void)
* apparent order.
*/
victim =
- pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);
+ pg_atomic_fetch_add_u32(&sweep->nextVictimBuffer, 1);
- if (victim >= NBuffers)
+ if (victim >= sweep->numBuffers)
{
uint32 originalVictim = victim;
/* always wrap what we look up in BufferDescriptors */
- victim = victim % NBuffers;
+ victim = victim % sweep->numBuffers;
/*
* If we're the one that just caused a wraparound, force
@@ -189,19 +213,23 @@ ClockSweepTick(void)
* could lead to an overflow of nextVictimBuffers, but that's
* highly unlikely and wouldn't be particularly harmful.
*/
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&sweep->clock_sweep_lock);
- wrapped = expected % NBuffers;
+ wrapped = expected % sweep->numBuffers;
- success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
+ success = pg_atomic_compare_exchange_u32(&sweep->nextVictimBuffer,
&expected, wrapped);
if (success)
- StrategyControl->completePasses++;
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ sweep->completePasses++;
+ SpinLockRelease(&sweep->clock_sweep_lock);
}
}
}
- return victim;
+
+ /* XXX buffer IDs are 1-based, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + (victim % sweep->numBuffers)));
+
+ return sweep->firstBuffer + victim;
}
static int
@@ -258,6 +286,28 @@ calculate_partition_index()
return index;
}
+/*
+ * ChooseClockSweep
+ * pick a clocksweep partition based on NUMA node and CPU
+ *
+ * The number of clocksweep partitions may not match the number of NUMA
+ * nodes, but it should not be lower. Each partition should be mapped to
+ * a single NUMA node, but a node may have multiple partitions. If there
+ * are multiple partitions per node (all nodes have the same number of
+ * partitions), we pick the partition using CPU.
+ *
+ * XXX Maybe we should do both the total and "per group" counts a power of
+ * two? That'd allow using shifts instead of divisions in the calculation,
+ * and that's cheaper. But how would that deal with odd number of nodes?
+ */
+static ClockSweep *
+ChooseClockSweep(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->sweeps[index];
+}
+
/*
* ChooseFreeList
* Pick the buffer freelist to use, depending on the CPU and NUMA node.
@@ -374,7 +424,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
+ pg_atomic_fetch_add_u32(&ChooseClockSweep()->numBufferAllocs, 1);
/*
* First check, without acquiring the lock, whether there's buffers in the
@@ -445,13 +495,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
/*
* Nothing on the freelist, so run the "clock sweep" algorithm
*
- * XXX Should we also make this NUMA-aware, to only access buffers from
- * the same NUMA node? That'd probably mean we need to make the clock
- * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
- * subset of buffers. But that also means each process could "sweep" only
- * a fraction of buffers, even if the other buffers are better candidates
- * for eviction. Would that also mean we'd have multiple bgwriters, one
- * for each node, or would one bgwriter handle all of that?
+ * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
+ * buffers from a single partition, aligned with the NUMA node. That means
+ * it only accesses buffers from the same NUMA node.
+ *
+ * XXX That also means each process "sweeps" only a fraction of buffers,
+ * even if the other buffers are better candidates for eviction. Maybe
+ * there should be some logic to "steal" buffers from other freelists or
+ * other nodes?
+ *
+ * XXX Would that also mean we'd have multiple bgwriters, one for each
+ * node, or would one bgwriter handle all of that?
*/
trycounter = NBuffers;
for (;;)
@@ -533,6 +587,41 @@ StrategyFreeBuffer(BufferDesc *buf)
SpinLockRelease(&freelist->freelist_lock);
}
+/*
+ * StrategySyncStart -- prepare for sync of all partitions
+ *
+ * Determine the number of clocksweep partitions, and calculate the recent
+ * buffers allocs (as a sum of all the partitions). This allows BgBufferSync
+ * to calculate average number of allocations per partition for the next
+ * sync cycle.
+ *
+ * In addition it returns the count of recent buffer allocs, which is a total
+ * summed from all partitions. The alloc counts are reset after being read,
+ * as the partitions are walked.
+ */
+void
+StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc)
+{
+ *num_buf_alloc = 0;
+ *num_parts = StrategyControl->num_partitions;
+
+ /*
+ * We lock the partitions one by one, so not exacly in sync, but that
+ * should be fine. We're only looking for heuristics anyway.
+ */
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ ClockSweep *sweep = &StrategyControl->sweeps[i];
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ if (num_buf_alloc)
+ {
+ *num_buf_alloc += pg_atomic_exchange_u32(&sweep->numBufferAllocs, 0);
+ }
+ SpinLockRelease(&sweep->clock_sweep_lock);
+ }
+}
+
/*
* StrategySyncStart -- tell BgBufferSync where to start syncing
*
@@ -540,37 +629,44 @@ StrategyFreeBuffer(BufferDesc *buf)
* BgBufferSync() will proceed circularly around the buffer array from there.
*
* In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * the higher-order bits of nextVictimBuffer).
+ *
+ * This only considers a single clocksweep partition, as BgBufferSync looks
+ * at them one by one.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers)
{
uint32 nextVictimBuffer;
int result;
+ ClockSweep *sweep = &StrategyControl->sweeps[partition];
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
- nextVictimBuffer = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer);
- result = nextVictimBuffer % NBuffers;
+ Assert((partition >= 0) && (partition < StrategyControl->num_partitions));
+
+ SpinLockAcquire(&sweep->clock_sweep_lock);
+ nextVictimBuffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+ result = nextVictimBuffer % sweep->numBuffers;
+
+ *first_buffer = sweep->firstBuffer;
+ *num_buffers = sweep->numBuffers;
if (complete_passes)
{
- *complete_passes = StrategyControl->completePasses;
+ *complete_passes = sweep->completePasses;
/*
* Additionally add the number of wraparounds that happened before
* completePasses could be incremented. C.f. ClockSweepTick().
*/
- *complete_passes += nextVictimBuffer / NBuffers;
+ *complete_passes += nextVictimBuffer / sweep->numBuffers;
}
+ SpinLockRelease(&sweep->clock_sweep_lock);
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
- }
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
- return result;
+ /* XXX buffer IDs start at 1, we're calculating 0-based indexes */
+ Assert(BufferIsValid(1 + sweep->firstBuffer + result));
+
+ return sweep->firstBuffer + result;
}
/*
@@ -658,6 +754,10 @@ StrategyShmemSize(void)
size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
num_partitions)));
+ /* size of clocksweep partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(ClockSweep),
+ num_partitions)));
+
return size;
}
@@ -676,6 +776,7 @@ StrategyInitialize(bool init)
int num_nodes;
int num_partitions;
int num_partitions_per_node;
+ char *ptr;
/* */
BufferPartitionParams(&num_partitions, &num_nodes);
@@ -703,7 +804,8 @@ StrategyInitialize(bool init)
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
- MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions) +
+ MAXALIGN(sizeof(ClockSweep) * num_partitions),
&found);
if (!found)
@@ -718,12 +820,41 @@ StrategyInitialize(bool init)
SpinLockInit(&StrategyControl->buffer_strategy_lock);
- /* Initialize the clock sweep pointer */
- pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
+ /* have to point the sweeps array to right after the freelists */
+ ptr = (char *) StrategyControl +
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions);
+ StrategyControl->sweeps = (ClockSweep *) ptr;
+
+ /* Initialize the clock sweep pointers (for all partitions) */
+ for (int i = 0; i < num_partitions; i++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ SpinLockInit(&StrategyControl->sweeps[i].clock_sweep_lock);
+
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].nextVictimBuffer, 0);
- /* Clear statistics */
- StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ /* get info about the buffer partition */
+ BufferPartitionGet(i, &node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ /*
+ * FIXME This may not quite right, because if NBuffers is not a
+ * perfect multiple of numBuffers, the last partition will have
+ * numBuffers set too high. buf_init handles this by tracking the
+ * remaining number of buffers, and not overflowing.
+ */
+ StrategyControl->sweeps[i].numBuffers = num_buffers;
+ StrategyControl->sweeps[i].firstBuffer = first_buffer;
+
+ /* Clear statistics */
+ StrategyControl->sweeps[i].completePasses = 0;
+ pg_atomic_init_u32(&StrategyControl->sweeps[i].numBufferAllocs, 0);
+ }
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -771,7 +902,6 @@ StrategyInitialize(bool init)
buf->freeNext = freelist->firstFreeBuffer;
freelist->firstFreeBuffer = i;
}
-
}
}
else
@@ -1111,9 +1241,11 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
}
void
-FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free,
+ uint32 *complete_passes, uint32 *buffer_allocs, uint32 *next_victim_buffer)
{
BufferStrategyFreelist *freelist;
+ ClockSweep *sweep;
int cur;
/* stats */
@@ -1123,6 +1255,7 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
freelist = &StrategyControl->freelists[idx];
+ sweep = &StrategyControl->sweeps[idx];
/* stat */
SpinLockAcquire(&freelist->freelist_lock);
@@ -1152,4 +1285,11 @@ FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actu
*remain = cnt_remain;
*actually_free = cnt_free;
+
+ /* get the clocksweep stats too */
+ *complete_passes = sweep->completePasses;
+ *buffer_allocs = pg_atomic_read_u32(&sweep->numBufferAllocs);
+ *next_victim_buffer = pg_atomic_read_u32(&sweep->nextVictimBuffer);
+
+ *next_victim_buffer = sweep->firstBuffer + (*next_victim_buffer % sweep->numBuffers);
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9dfbecb9fe4..907b160b4f7 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -449,7 +449,9 @@ extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf, bool from_ring);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategySyncPrepare(int *num_parts, uint32 *num_buf_alloc);
+extern int StrategySyncStart(int partition, uint32 *complete_passes,
+ int *first_buffer, int *num_buffers);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index df127274190..53855d4be23 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -349,7 +349,10 @@ extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
extern void FreelistPartitionGetInfo(int idx,
uint64 *consumed, uint64 *remain,
- uint64 *actually_free);
+ uint64 *actually_free,
+ uint32 *complete_passes,
+ uint32 *buffer_allocs,
+ uint32 *next_victim_buffer);
/* inline functions */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c695cfa76e8..a38dd8d6242 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ ClientCertName
ClientConnectionInfo
ClientData
ClientSocket
+ClockSweep
ClonePtrType
ClosePortalStmt
ClosePtrType
--
2.50.1
[text/x-patch] v3-0004-NUMA-partition-buffer-freelist.patch (20.9K, ../[email protected]/5-v3-0004-NUMA-partition-buffer-freelist.patch)
download | inline diff:
From 06a43d54498ab5049c12a458cfbd4fe3b3b168c2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:38:41 +0200
Subject: [PATCH v3 4/7] NUMA: partition buffer freelist
Instead of a single buffer freelist, partition into multiple smaller
lists, to reduce lock contention, and to spread the buffers over all
NUMA nodes more evenly.
This uses the buffer partitioning scheme introduced by the earlier
patch, i.e. the partitions will "align" with NUMA nodes, etc.
It also extends the "pg_buffercache_partitions" view, to include
information about each freelist (number of consumedd buffers, ...).
When allocating a buffer, it's taken from the correct freelist (same
NUMA node).
Note: This is (probably) more important than partitioning ProcArray.
---
.../pg_buffercache--1.6--1.7.sql | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 24 +-
src/backend/storage/buffer/freelist.c | 360 ++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/bufmgr.h | 4 +-
7 files changed, 372 insertions(+), 30 deletions(-)
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
index bd97246f6ab..3871c261528 100644
--- a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -12,7 +12,7 @@ LANGUAGE C PARALLEL SAFE;
-- Create a view for convenient access.
CREATE VIEW pg_buffercache_partitions AS
SELECT P.* FROM pg_buffercache_partitions() AS P
- (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer);
+ (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer, buffers_consumed bigint, buffers_remain bigint, buffers_free bigint);
-- Don't want these to be available to public.
REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index 8baa7c7b543..668ada8c47b 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,7 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
-#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 8
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -812,6 +812,12 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
INT4OID, -1, 0);
TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 6, "buffers_consumed",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 7, "buffers_remain",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 8, "buffers_free",
+ INT8OID, -1, 0);
funcctx->user_fctx = BlessTupleDesc(tupledesc);
@@ -833,12 +839,19 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
first_buffer,
last_buffer;
+ uint64 buffers_consumed,
+ buffers_remain,
+ buffers_free;
+
Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
BufferPartitionGet(i, &numa_node, &num_buffers,
&first_buffer, &last_buffer);
+ FreelistPartitionGetInfo(i, &buffers_consumed, &buffers_remain,
+ &buffers_free);
+
values[0] = Int32GetDatum(i);
nulls[0] = false;
@@ -854,6 +867,15 @@ pg_buffercache_partitions(PG_FUNCTION_ARGS)
values[4] = Int32GetDatum(last_buffer);
nulls[4] = false;
+ values[5] = Int64GetDatum(buffers_consumed);
+ nulls[5] = false;
+
+ values[6] = Int64GetDatum(buffers_remain);
+ nulls[6] = false;
+
+ values[7] = Int64GetDatum(buffers_free);
+ nulls[7] = false;
+
/* Build and return the tuple. */
tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
result = HeapTupleGetDatum(tuple);
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index e046526c149..c3fbd651dd5 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,14 +15,52 @@
*/
#include "postgres.h"
+#include <sched.h>
+#include <sys/sysinfo.h>
+
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/ipc.h"
#include "storage/proc.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
+/*
+ * Represents one freelist partition.
+ */
+typedef struct BufferStrategyFreelist
+{
+ /* Spinlock: protects the values below */
+ slock_t freelist_lock;
+
+ /*
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres.
+ */
+ int firstFreeBuffer __attribute__((aligned(64))); /* Head of list of
+ * unused buffers */
+
+ /* Number of buffers consumed from this list. */
+ uint64 consumed;
+} BufferStrategyFreelist;
+
+/*
+ * The minimum number of partitions we want to have. We want at least this
+ * number of partitions, even on non-NUMA system, as it helps with contention
+ * for buffers. But with multiple NUMA nodes, we want a separate partition per
+ * node. But we may get multiple partitions per node, for low node count.
+ *
+ * With multiple partitions per NUMA node, we pick the partition based on CPU
+ * (or some other parameter).
+ */
+#define MIN_FREELIST_PARTITIONS 4
/*
* The shared freelist control information.
@@ -39,8 +77,6 @@ typedef struct
*/
pg_atomic_uint32 nextVictimBuffer;
- int firstFreeBuffer; /* Head of list of unused buffers */
-
/*
* Statistics. These counters should be wide enough that they can't
* overflow during a single bgwriter cycle.
@@ -51,8 +87,19 @@ typedef struct
/*
* Bgworker process to be notified upon activity or -1 if none. See
* StrategyNotifyBgWriter.
+ *
+ * XXX Not sure why this needs to be aligned like this. Need to ask
+ * Andres. Also, shouldn't the alignment be specified after, like for
+ * "consumed"?
*/
- int bgwprocno;
+ int __attribute__((aligned(64))) bgwprocno;
+
+ /* info about freelist partitioning */
+ int num_nodes; /* effectively number of NUMA nodes */
+ int num_partitions;
+ int num_partitions_per_node;
+
+ BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
} BufferStrategyControl;
/* Pointers to shared state */
@@ -157,6 +204,88 @@ ClockSweepTick(void)
return victim;
}
+static int
+calculate_partition_index()
+{
+ int rc;
+ unsigned cpu;
+ unsigned node;
+ int index;
+
+ Assert(StrategyControl->num_partitions ==
+ (StrategyControl->num_nodes * StrategyControl->num_partitions_per_node));
+
+ /*
+ * freelist is partitioned, so determine the CPU/NUMA node, and pick a
+ * list based on that.
+ */
+ rc = getcpu(&cpu, &node);
+ if (rc != 0)
+ elog(ERROR, "getcpu failed: %m");
+
+ /*
+ * XXX We should't get nodes that we haven't considered while building the
+ * partitions. Maybe if we allow this (e.g. due to support adjusting the
+ * NUMA stuff at runtime), we should just do our best to minimize the
+ * conflicts somehow. But it'll make the mapping harder, so for now we
+ * ignore it.
+ */
+ if (node > StrategyControl->num_nodes)
+ elog(ERROR, "node out of range: %d > %u", cpu, StrategyControl->num_nodes);
+
+ /*
+ * Find the partition. If we have a single partition per node, we can
+ * calculate the index directly from node. Otherwise we need to do two
+ * steps, using node and then cpu.
+ */
+ if (StrategyControl->num_partitions_per_node == 1)
+ {
+ index = (node % StrategyControl->num_partitions);
+ }
+ else
+ {
+ int index_group,
+ index_part;
+
+ /* two steps - calculate group from node, partition from cpu */
+ index_group = (node % StrategyControl->num_nodes);
+ index_part = (cpu % StrategyControl->num_partitions_per_node);
+
+ index = (index_group * StrategyControl->num_partitions_per_node)
+ + index_part;
+ }
+
+ return index;
+}
+
+/*
+ * ChooseFreeList
+ * Pick the buffer freelist to use, depending on the CPU and NUMA node.
+ *
+ * Without partitioned freelists (numa_partition_freelist=false), there's only
+ * a single freelist, so use that.
+ *
+ * With partitioned freelists, we have multiple ways how to pick the freelist
+ * for the backend:
+ *
+ * - one freelist per CPU, use the freelist for CPU the task executes on
+ *
+ * - one freelist per NUMA node, use the freelist for node task executes on
+ *
+ * - use fixed number of freelists, map processes to lists based on PID
+ *
+ * There may be some other strategies, not sure. The important thing is this
+ * needs to be refrecled during initialization, i.e. we need to create the
+ * right number of lists.
+ */
+static BufferStrategyFreelist *
+ChooseFreeList(void)
+{
+ int index = calculate_partition_index();
+
+ return &StrategyControl->freelists[index];
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -168,10 +297,13 @@ ClockSweepTick(void)
bool
have_free_buffer(void)
{
- if (StrategyControl->firstFreeBuffer >= 0)
- return true;
- else
- return false;
+ for (int i = 0; i < StrategyControl->num_partitions; i++)
+ {
+ if (StrategyControl->freelists[i].firstFreeBuffer >= 0)
+ return true;
+ }
+
+ return false;
}
/*
@@ -193,6 +325,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ BufferStrategyFreelist *freelist;
*from_ring = false;
@@ -259,31 +392,35 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
* buffer_strategy_lock not the individual buffer spinlocks, so it's OK to
* manipulate them without holding the spinlock.
*/
- if (StrategyControl->firstFreeBuffer >= 0)
+ freelist = ChooseFreeList();
+ if (freelist->firstFreeBuffer >= 0)
{
while (true)
{
/* Acquire the spinlock to remove element from the freelist */
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ SpinLockAcquire(&freelist->freelist_lock);
- if (StrategyControl->firstFreeBuffer < 0)
+ if (freelist->firstFreeBuffer < 0)
{
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
break;
}
- buf = GetBufferDescriptor(StrategyControl->firstFreeBuffer);
+ buf = GetBufferDescriptor(freelist->firstFreeBuffer);
Assert(buf->freeNext != FREENEXT_NOT_IN_LIST);
/* Unconditionally remove buffer from freelist */
- StrategyControl->firstFreeBuffer = buf->freeNext;
+ freelist->firstFreeBuffer = buf->freeNext;
buf->freeNext = FREENEXT_NOT_IN_LIST;
+ /* increment number of buffers we consumed from this list */
+ freelist->consumed++;
+
/*
* Release the lock so someone else can access the freelist while
* we check out this buffer.
*/
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
/*
* If the buffer is pinned or has a nonzero usage_count, we cannot
@@ -305,7 +442,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ /*
+ * Nothing on the freelist, so run the "clock sweep" algorithm
+ *
+ * XXX Should we also make this NUMA-aware, to only access buffers from
+ * the same NUMA node? That'd probably mean we need to make the clock
+ * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
+ * subset of buffers. But that also means each process could "sweep" only
+ * a fraction of buffers, even if the other buffers are better candidates
+ * for eviction. Would that also mean we'd have multiple bgwriters, one
+ * for each node, or would one bgwriter handle all of that?
+ */
trycounter = NBuffers;
for (;;)
{
@@ -356,7 +503,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
void
StrategyFreeBuffer(BufferDesc *buf)
{
- SpinLockAcquire(&StrategyControl->buffer_strategy_lock);
+ BufferStrategyFreelist *freelist;
+
+ /*
+ * We don't want to call ChooseFreeList() again, because we might get a
+ * completely different freelist - either a different partition in the
+ * same group, or even a different group if the NUMA node changed. But we
+ * can calculate the proper freelist from the buffer id.
+ */
+ int index = (BufferGetNode(buf->buf_id) * StrategyControl->num_partitions_per_node)
+ + (buf->buf_id % StrategyControl->num_partitions_per_node);
+
+ Assert((index >= 0) && (index < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[index];
+
+ SpinLockAcquire(&freelist->freelist_lock);
/*
* It is possible that we are told to put something in the freelist that
@@ -364,11 +526,11 @@ StrategyFreeBuffer(BufferDesc *buf)
*/
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
- buf->freeNext = StrategyControl->firstFreeBuffer;
- StrategyControl->firstFreeBuffer = buf->buf_id;
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = buf->buf_id;
}
- SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+ SpinLockRelease(&freelist->freelist_lock);
}
/*
@@ -432,6 +594,42 @@ StrategyNotifyBgWriter(int bgwprocno)
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
}
+/* prints some debug info / stats about freelists at shutdown */
+static void
+freelist_before_shmem_exit(int code, Datum arg)
+{
+ for (int p = 0; p < StrategyControl->num_partitions; p++)
+ {
+ BufferStrategyFreelist *freelist = &StrategyControl->freelists[p];
+ uint64 remain = 0;
+ uint64 actually_free = 0;
+ int cur = freelist->firstFreeBuffer;
+
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ actually_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ elog(LOG, "NUMA: freelist partition %d, firstF: %d: consumed: %lu, remain: %lu, actually free: %lu",
+ p,
+ freelist->firstFreeBuffer,
+ freelist->consumed,
+ remain, actually_free);
+ }
+}
/*
* StrategyShmemSize
@@ -445,12 +643,20 @@ Size
StrategyShmemSize(void)
{
Size size = 0;
+ int num_partitions;
+ int num_nodes;
+
+ BufferPartitionParams(&num_partitions, &num_nodes);
/* size of lookup hash table ... see comment in StrategyInitialize */
size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
/* size of the shared replacement strategy control block */
- size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
+ size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
+
+ /* size of freelist partitions (at least one per NUMA node) */
+ size = add_size(size, MAXALIGN(mul_size(sizeof(BufferStrategyFreelist),
+ num_partitions)));
return size;
}
@@ -467,6 +673,18 @@ StrategyInitialize(bool init)
{
bool found;
+ int num_nodes;
+ int num_partitions;
+ int num_partitions_per_node;
+
+ /* */
+ BufferPartitionParams(&num_partitions, &num_nodes);
+
+ /* always a multiple of NUMA nodes */
+ Assert(num_partitions % num_nodes == 0);
+
+ num_partitions_per_node = (num_partitions / num_nodes);
+
/*
* Initialize the shared buffer lookup hashtable.
*
@@ -484,7 +702,8 @@ StrategyInitialize(bool init)
*/
StrategyControl = (BufferStrategyControl *)
ShmemInitStruct("Buffer Strategy Status",
- sizeof(BufferStrategyControl),
+ MAXALIGN(offsetof(BufferStrategyControl, freelists)) +
+ MAXALIGN(sizeof(BufferStrategyFreelist) * num_partitions),
&found);
if (!found)
@@ -494,13 +713,10 @@ StrategyInitialize(bool init)
*/
Assert(init);
- SpinLockInit(&StrategyControl->buffer_strategy_lock);
+ /* register callback to dump some stats on exit */
+ before_shmem_exit(freelist_before_shmem_exit, 0);
- /*
- * Grab the whole linked list of free buffers for our strategy. We
- * assume it was previously set up by BufferManagerShmemInit().
- */
- StrategyControl->firstFreeBuffer = 0;
+ SpinLockInit(&StrategyControl->buffer_strategy_lock);
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
@@ -511,6 +727,52 @@ StrategyInitialize(bool init)
/* No pending notification */
StrategyControl->bgwprocno = -1;
+
+ /* initialize the partitioned clocksweep */
+ StrategyControl->num_partitions = num_partitions;
+ StrategyControl->num_nodes = num_nodes;
+ StrategyControl->num_partitions_per_node = num_partitions_per_node;
+
+ /*
+ * Rebuild the freelist - right now all buffers are in one huge list,
+ * we want to rework that into multiple lists. Start by initializing
+ * the strategy to have empty lists.
+ */
+ for (int nfreelist = 0; nfreelist < num_partitions; nfreelist++)
+ {
+ int node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ BufferStrategyFreelist *freelist;
+
+ freelist = &StrategyControl->freelists[nfreelist];
+
+ freelist->firstFreeBuffer = FREENEXT_END_OF_LIST;
+
+ SpinLockInit(&freelist->freelist_lock);
+
+ /* get info about the buffer partition */
+ BufferPartitionGet(nfreelist, &node,
+ &num_buffers, &first_buffer, &last_buffer);
+
+ /*
+ * Walk through buffers for each partition, add them to the list.
+ * Walk from the end, because we're adding the buffers to the
+ * beginning.
+ */
+
+ for (int i = last_buffer; i >= first_buffer; i--)
+ {
+ BufferDesc *buf = GetBufferDescriptor(i);
+
+ /* add to the freelist */
+ buf->freeNext = freelist->firstFreeBuffer;
+ freelist->firstFreeBuffer = i;
+ }
+
+ }
}
else
Assert(!init);
@@ -847,3 +1109,47 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_r
return true;
}
+
+void
+FreelistPartitionGetInfo(int idx, uint64 *consumed, uint64 *remain, uint64 *actually_free)
+{
+ BufferStrategyFreelist *freelist;
+ int cur;
+
+ /* stats */
+ uint64 cnt_remain = 0;
+ uint64 cnt_free = 0;
+
+ Assert((idx >= 0) && (idx < StrategyControl->num_partitions));
+
+ freelist = &StrategyControl->freelists[idx];
+
+ /* stat */
+ SpinLockAcquire(&freelist->freelist_lock);
+
+ *consumed = freelist->consumed;
+
+ cur = freelist->firstFreeBuffer;
+ while (cur >= 0)
+ {
+ uint32 local_buf_state;
+ BufferDesc *buf;
+
+ buf = GetBufferDescriptor(cur);
+
+ cnt_remain++;
+
+ local_buf_state = LockBufHdr(buf);
+
+ if (!(local_buf_state & BM_TAG_VALID))
+ cnt_free++;
+
+ UnlockBufHdr(buf, local_buf_state);
+
+ cur = buf->freeNext;
+ }
+ SpinLockRelease(&freelist->freelist_lock);
+
+ *remain = cnt_remain;
+ *actually_free = cnt_free;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index f5359db3656..a11bc71a386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -148,6 +148,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
bool numa_localalloc = false;
+bool numa_partition_freelist = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a21f20800fb..0552ed62cc7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2136,6 +2136,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_partition_freelist", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables buffer freelists to be partitioned per NUMA node."),
+ gettext_noop("When enabled, we create a separate freelist per NUMA node."),
+ },
+ &numa_partition_freelist,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 692871a401f..66baf2bf33e 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -180,6 +180,7 @@ extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
extern PGDLLIMPORT bool numa_localalloc;
+extern PGDLLIMPORT bool numa_partition_freelist;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index deaf4f19fa4..df127274190 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -347,7 +347,9 @@ extern int GetAccessStrategyBufferCount(BufferAccessStrategy strategy);
extern int GetAccessStrategyPinLimit(BufferAccessStrategy strategy);
extern void FreeAccessStrategy(BufferAccessStrategy strategy);
-
+extern void FreelistPartitionGetInfo(int idx,
+ uint64 *consumed, uint64 *remain,
+ uint64 *actually_free);
/* inline functions */
--
2.50.1
[text/x-patch] v3-0003-freelist-Don-t-track-tail-of-a-freelist.patch (1.6K, ../[email protected]/6-v3-0003-freelist-Don-t-track-tail-of-a-freelist.patch)
download | inline diff:
From 3c7dbbec4ee3957c92bf647605768beb5473f66b Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Mon, 14 Oct 2024 14:10:13 -0400
Subject: [PATCH v3 3/7] freelist: Don't track tail of a freelist
The freelist tail isn't currently used, making it unnecessary overhead.
So just don't do that.
---
src/backend/storage/buffer/freelist.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 01909be0272..e046526c149 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -40,12 +40,6 @@ typedef struct
pg_atomic_uint32 nextVictimBuffer;
int firstFreeBuffer; /* Head of list of unused buffers */
- int lastFreeBuffer; /* Tail of list of unused buffers */
-
- /*
- * NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,
- * when the list is empty)
- */
/*
* Statistics. These counters should be wide enough that they can't
@@ -371,8 +365,6 @@ StrategyFreeBuffer(BufferDesc *buf)
if (buf->freeNext == FREENEXT_NOT_IN_LIST)
{
buf->freeNext = StrategyControl->firstFreeBuffer;
- if (buf->freeNext < 0)
- StrategyControl->lastFreeBuffer = buf->buf_id;
StrategyControl->firstFreeBuffer = buf->buf_id;
}
@@ -509,7 +501,6 @@ StrategyInitialize(bool init)
* assume it was previously set up by BufferManagerShmemInit().
*/
StrategyControl->firstFreeBuffer = 0;
- StrategyControl->lastFreeBuffer = NBuffers - 1;
/* Initialize the clock sweep pointer */
pg_atomic_init_u32(&StrategyControl->nextVictimBuffer, 0);
--
2.50.1
[text/x-patch] v3-0002-NUMA-localalloc.patch (3.7K, ../[email protected]/7-v3-0002-NUMA-localalloc.patch)
download | inline diff:
From d08a94b0f54a4c4986d351f98269905bb511624c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 22 May 2025 18:27:06 +0200
Subject: [PATCH v3 2/7] NUMA: localalloc
Set the default allocation policy to "localalloc", which means from the
local NUMA node. This is useful for process-private memory, which is not
going to be shared with other nodes, and is relatively short-lived (so
we're unlikely to have issues if the process gets moved by scheduler).
This sets default for the whole process, for all future allocations. But
that's fine, we've already populated the shared memory earlier (by
interleaving it explicitly). Otherwise we'd trigger page fault and it'd
be allocated on local node.
XXX This patch may not be necessary, as we now locate memory to nodes
using explicit numa_tonode_memory() calls, and not by interleaving. But
it's useful for experiments during development, so I'm keeping it.
---
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 17 +++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 ++++++++++
src/include/miscadmin.h | 1 +
4 files changed, 29 insertions(+)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 876cb64cf66..f5359db3656 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -147,6 +147,7 @@ int MaxBackends = 0;
/* NUMA stuff */
bool numa_buffers_interleave = false;
+bool numa_localalloc = false;
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 43b4dbccc3d..079974944e9 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -28,6 +28,10 @@
#include <arpa/inet.h>
#include <utime.h>
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#endif
+
#include "access/htup_details.h"
#include "access/parallel.h"
#include "catalog/pg_authid.h"
@@ -164,6 +168,19 @@ InitPostmasterChild(void)
(errcode_for_socket_access(),
errmsg_internal("could not set postmaster death monitoring pipe to FD_CLOEXEC mode: %m")));
#endif
+
+#ifdef USE_LIBNUMA
+
+ /*
+ * Set the default allocation policy to local node, where the task is
+ * executing at the time of a page fault.
+ *
+ * XXX I believe this is not necessary, now that we don't use automatic
+ * interleaving (numa_set_interleave_mask).
+ */
+ if (numa_localalloc)
+ numa_set_localalloc();
+#endif
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9570087aa60..a21f20800fb 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2126,6 +2126,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_localalloc", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables setting the default allocation policy to local node."),
+ gettext_noop("When enabled, allocate from the node where the task is executing."),
+ },
+ &numa_localalloc,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 014a6079af2..692871a401f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -179,6 +179,7 @@ extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
extern PGDLLIMPORT bool numa_buffers_interleave;
+extern PGDLLIMPORT bool numa_localalloc;
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
--
2.50.1
[text/x-patch] v3-0001-NUMA-interleaving-buffers.patch (38.2K, ../[email protected]/8-v3-0001-NUMA-interleaving-buffers.patch)
download | inline diff:
From f0eb1af6fdcfd7daae26952ddc223952333f6af2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Mon, 28 Jul 2025 14:01:37 +0200
Subject: [PATCH v3 1/7] NUMA: interleaving buffers
Ensure shared buffers are allocated from all NUMA nodes, in a balanced
way, instead of just using the node where Postgres initially starts, or
where the kernel decides to migrate the page, etc. With pre-warming
performed by a single backend, this can easily result in severely
unbalanced memory distribution (with most from a single NUMA node).
The kernel would eventually move some of the memory to other nodes
(thanks to zone_reclaim), but that tends to take a long time. So this
patch improves predictability, reduces the time needed for warmup
during benchmarking, etc. It's less dependent on what the CPU
scheduler does, etc.
Furthermore, the buffers are mapped to NUMA nodes in a deterministic
way, so this also allows further improvements like backends using
buffers from the same NUMA node.
The effect is similar to
numactl --interleave=all
but there's a number of important differences.
Firstly, it's applied only to shared buffers (and also to descriptors),
not to the whole shared memory segment. It's not clear we'd want to use
interleaving for all parts, storing entries with different sizes and
life cycles (e.g. ProcArray may need different approach).
Secondly, it considers the page and block size, and makes sure to always
put the whole buffer on a single NUMA node (even if it happens to use
multiple memory pages), and to keep the buffer and it's descriptor on
the same NUMA node. The seriousness/likelihood of these issues depends
on the memory page size (regular vs. huge pages).
The mapping of memory to NUMA nodes happens in larger chunks. This is
required to handle buffer descriptors (which are smaller than buffers),
and so many more fit onto a single memory page.
The number of buffer descriptors per memory page determines the smallest
number of buffers that can be placed on a NUMA node. With 2MB huge pages
this is 256MB, with 4KB pages this is 512KB). Nodes get a multiple of
this, and we try to keep the nodes balanced - the last node can get less
memory, though.
The "buffer partitions" may not be 1:1 with NUMA nodes. There's a
minimal number of partitions (default: 4) that will be created even with
fewer NUMA nodes, or no NUMA at all. Each node gets the same number of
partitions, to keep things simple. For example, with 2 nodes there'll be
4 partitions, with each node getting 2 of them. With 3 nodes there'll be
6 partitions (again, 2 per node).
The patch introduces a simple "registry" of buffer partitions, keeping
track of the first/last buffer, NUMA node, etc. This serves as a source
of truth, both for this patch and for later patches building on this
same buffer partition structure.
With the feature disabled (GUC set to 'off'), there'll be a single
partition for all the buffers (and it won't be mapped to a NUMA node).
Notes:
* The feature is enabled by numa_buffers_interleave GUC (default: false)
* It's not clear we want to enable interleaving for all shared memory.
We probably want that for shared buffers, but maybe not for ProcArray
or freelists.
* Similar questions are about huge pages - in general it's a good idea,
but maybe it's not quite good for ProcArray. It's somewhate separate
from NUMA, but not entirely because NUMA works on page granularity.
PGPROC entries are ~8KB, so too large for interleaving with 4K pages,
as we don't want to split the entry to multiple nodes. But could be
done explicitly, by specifying which node to use for the pages.
* We could partition ProcArray, with one partition per NUMA node, and
then at connection time pick a node from the same node. The process
could migrate to some other node later, especially for long-lived
connections, but there's no perfect solution, Maybe we could set
affinity to cores from the same node, or something like that?
---
contrib/pg_buffercache/Makefile | 2 +-
.../pg_buffercache--1.6--1.7.sql | 22 +
contrib/pg_buffercache/pg_buffercache.control | 2 +-
contrib/pg_buffercache/pg_buffercache_pages.c | 92 +++
src/backend/storage/buffer/buf_init.c | 626 +++++++++++++++++-
src/backend/utils/init/globals.c | 3 +
src/backend/utils/misc/guc_tables.c | 10 +
src/include/miscadmin.h | 2 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 15 +
src/tools/pgindent/typedefs.list | 2 +
11 files changed, 771 insertions(+), 11 deletions(-)
create mode 100644 contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
index 5f748543e2e..0e618f66aec 100644
--- a/contrib/pg_buffercache/Makefile
+++ b/contrib/pg_buffercache/Makefile
@@ -9,7 +9,7 @@ EXTENSION = pg_buffercache
DATA = pg_buffercache--1.2.sql pg_buffercache--1.2--1.3.sql \
pg_buffercache--1.1--1.2.sql pg_buffercache--1.0--1.1.sql \
pg_buffercache--1.3--1.4.sql pg_buffercache--1.4--1.5.sql \
- pg_buffercache--1.5--1.6.sql
+ pg_buffercache--1.5--1.6.sql pg_buffercache--1.6--1.7.sql
PGFILEDESC = "pg_buffercache - monitoring of shared buffer cache in real-time"
REGRESS = pg_buffercache pg_buffercache_numa
diff --git a/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
new file mode 100644
index 00000000000..bd97246f6ab
--- /dev/null
+++ b/contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql
@@ -0,0 +1,22 @@
+/* contrib/pg_buffercache/pg_buffercache--1.6--1.7.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION pg_buffercache UPDATE TO '1.7'" to load this file. \quit
+
+-- Register the new functions.
+CREATE OR REPLACE FUNCTION pg_buffercache_partitions()
+RETURNS SETOF RECORD
+AS 'MODULE_PATHNAME', 'pg_buffercache_partitions'
+LANGUAGE C PARALLEL SAFE;
+
+-- Create a view for convenient access.
+CREATE VIEW pg_buffercache_partitions AS
+ SELECT P.* FROM pg_buffercache_partitions() AS P
+ (partition integer, numa_node integer, num_buffers integer, first_buffer integer, last_buffer integer);
+
+-- Don't want these to be available to public.
+REVOKE ALL ON FUNCTION pg_buffercache_partitions() FROM PUBLIC;
+REVOKE ALL ON pg_buffercache_partitions FROM PUBLIC;
+
+GRANT EXECUTE ON FUNCTION pg_buffercache_partitions() TO pg_monitor;
+GRANT SELECT ON pg_buffercache_partitions TO pg_monitor;
diff --git a/contrib/pg_buffercache/pg_buffercache.control b/contrib/pg_buffercache/pg_buffercache.control
index b030ba3a6fa..11499550945 100644
--- a/contrib/pg_buffercache/pg_buffercache.control
+++ b/contrib/pg_buffercache/pg_buffercache.control
@@ -1,5 +1,5 @@
# pg_buffercache extension
comment = 'examine the shared buffer cache'
-default_version = '1.6'
+default_version = '1.7'
module_pathname = '$libdir/pg_buffercache'
relocatable = true
diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c
index ae0291e6e96..8baa7c7b543 100644
--- a/contrib/pg_buffercache/pg_buffercache_pages.c
+++ b/contrib/pg_buffercache/pg_buffercache_pages.c
@@ -27,6 +27,7 @@
#define NUM_BUFFERCACHE_EVICT_ALL_ELEM 3
#define NUM_BUFFERCACHE_NUMA_ELEM 3
+#define NUM_BUFFERCACHE_PARTITIONS_ELEM 5
PG_MODULE_MAGIC_EXT(
.name = "pg_buffercache",
@@ -100,6 +101,7 @@ PG_FUNCTION_INFO_V1(pg_buffercache_usage_counts);
PG_FUNCTION_INFO_V1(pg_buffercache_evict);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_relation);
PG_FUNCTION_INFO_V1(pg_buffercache_evict_all);
+PG_FUNCTION_INFO_V1(pg_buffercache_partitions);
/* Only need to touch memory once per backend process lifetime */
@@ -771,3 +773,93 @@ pg_buffercache_evict_all(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * Inquire about partitioning of buffers between NUMA nodes.
+ */
+Datum
+pg_buffercache_partitions(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ MemoryContext oldcontext;
+ TupleDesc tupledesc;
+ TupleDesc expected_tupledesc;
+ HeapTuple tuple;
+ Datum result;
+
+ if (SRF_IS_FIRSTCALL())
+ {
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* Switch context when allocating stuff to be used in later calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ if (get_call_result_type(fcinfo, NULL, &expected_tupledesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (expected_tupledesc->natts != NUM_BUFFERCACHE_PARTITIONS_ELEM)
+ elog(ERROR, "incorrect number of output arguments");
+
+ /* Construct a tuple descriptor for the result rows. */
+ tupledesc = CreateTemplateTupleDesc(expected_tupledesc->natts);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 1, "partition",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 2, "numa_node",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 3, "num_buffers",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 4, "first_buffer",
+ INT4OID, -1, 0);
+ TupleDescInitEntry(tupledesc, (AttrNumber) 5, "last_buffer",
+ INT4OID, -1, 0);
+
+ funcctx->user_fctx = BlessTupleDesc(tupledesc);
+
+ /* Return to original context when allocating transient memory */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Set max calls and remember the user function context. */
+ funcctx->max_calls = BufferPartitionCount();
+ }
+
+ funcctx = SRF_PERCALL_SETUP();
+
+ if (funcctx->call_cntr < funcctx->max_calls)
+ {
+ uint32 i = funcctx->call_cntr;
+
+ int numa_node,
+ num_buffers,
+ first_buffer,
+ last_buffer;
+
+ Datum values[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+ bool nulls[NUM_BUFFERCACHE_PARTITIONS_ELEM];
+
+ BufferPartitionGet(i, &numa_node, &num_buffers,
+ &first_buffer, &last_buffer);
+
+ values[0] = Int32GetDatum(i);
+ nulls[0] = false;
+
+ values[1] = Int32GetDatum(numa_node);
+ nulls[1] = false;
+
+ values[2] = Int32GetDatum(num_buffers);
+ nulls[2] = false;
+
+ values[3] = Int32GetDatum(first_buffer);
+ nulls[3] = false;
+
+ values[4] = Int32GetDatum(last_buffer);
+ nulls[4] = false;
+
+ /* Build and return the tuple. */
+ tuple = heap_form_tuple((TupleDesc) funcctx->user_fctx, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1dc488a42..5b65a855b29 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,9 +14,17 @@
*/
#include "postgres.h"
+#ifdef USE_LIBNUMA
+#include <numa.h>
+#include <numaif.h>
+#endif
+
+#include "port/pg_numa.h"
#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/pg_shmem.h"
+#include "storage/proc.h"
BufferDescPadded *BufferDescriptors;
char *BufferBlocks;
@@ -24,6 +32,19 @@ ConditionVariableMinimallyPadded *BufferIOCVArray;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+BufferPartitions *BufferPartitionsArray;
+
+static Size get_memory_page_size(void);
+static void buffer_partitions_prepare(void);
+static void buffer_partitions_init(void);
+
+/* number of NUMA nodes (as returned by numa_num_configured_nodes) */
+static int numa_nodes = -1; /* number of nodes when sizing */
+static Size numa_page_size = 0; /* page used to size partitions */
+static bool numa_can_partition = false; /* can map to NUMA nodes? */
+static int numa_buffers_per_node = -1; /* buffers per node */
+static int numa_partitions = 0; /* total (multiple of nodes) */
+
/*
* Data Structures:
@@ -70,19 +91,89 @@ BufferManagerShmemInit(void)
bool foundBufs,
foundDescs,
foundIOCV,
- foundBufCkpt;
+ foundBufCkpt,
+ foundParts;
+ Size mem_page_size;
+ Size buffer_align;
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ /*
+ * With NUMA we need to ensure the buffers are properly aligned not just
+ * to PG_IO_ALIGN_SIZE, but also to memory page size, because NUMA works
+ * on page granularity, and we don't want a buffer to get split to
+ * multiple nodes (when using multiple memory pages).
+ *
+ * We also don't want to interfere with other parts of shared memory,
+ * which could easily happen with huge pages (e.g. with data stored before
+ * buffers).
+ *
+ * We do this by aligning to the larger of the two values (we know both
+ * are power-of-two values, so the larger value is automatically a
+ * multiple of the lesser one).
+ *
+ * XXX Maybe there's a way to use less alignment?
+ *
+ * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
+ * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
+ * that doesn't seem quite worth it. Maybe we should simply align to
+ * BLCKSZ, so that buffers don't get split? Still, we might interfere with
+ * other stuff stored in shared memory that we want to allocate on a
+ * particular NUMA node (e.g. ProcArray).
+ *
+ * XXX Maybe with "too large" huge pages we should just not do this, or
+ * maybe do this only for sufficiently large areas (e.g. shared buffers,
+ * but not ProcArray).
+ */
+ buffer_align = Max(mem_page_size, PG_IO_ALIGN_SIZE);
+
+ /* one page is a multiple of the other */
+ Assert(((mem_page_size % PG_IO_ALIGN_SIZE) == 0) ||
+ ((PG_IO_ALIGN_SIZE % mem_page_size) == 0));
+
+ /* allocate the partition registry first */
+ BufferPartitionsArray = (BufferPartitions *)
+ ShmemInitStruct("Buffer Partitions",
+ offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions),
+ &foundParts);
- /* Align descriptors to a cacheline boundary. */
+ /*
+ * Align descriptors to a cacheline boundary, and memory page.
+ *
+ * We want to distribute both to NUMA nodes, so that each buffer and it's
+ * descriptor are on the same NUMA node. So we align both the same way.
+ *
+ * XXX The memory page is always larger than cacheline, so the cacheline
+ * reference is a bit unnecessary.
+ *
+ * XXX In principle we only need to do this with NUMA, otherwise we could
+ * still align just to cacheline, as before.
+ */
BufferDescriptors = (BufferDescPadded *)
- ShmemInitStruct("Buffer Descriptors",
- NBuffers * sizeof(BufferDescPadded),
- &foundDescs);
+ TYPEALIGN(buffer_align,
+ ShmemInitStruct("Buffer Descriptors",
+ NBuffers * sizeof(BufferDescPadded) + buffer_align,
+ &foundDescs));
/* Align buffer pool on IO page size boundary. */
BufferBlocks = (char *)
- TYPEALIGN(PG_IO_ALIGN_SIZE,
+ TYPEALIGN(buffer_align,
ShmemInitStruct("Buffer Blocks",
- NBuffers * (Size) BLCKSZ + PG_IO_ALIGN_SIZE,
+ NBuffers * (Size) BLCKSZ + buffer_align,
&foundBufs));
/* Align condition variables to cacheline boundary. */
@@ -112,6 +203,12 @@ BufferManagerShmemInit(void)
{
int i;
+ /*
+ * Initialize the registry of buffer partitions, and also move the
+ * memory to different NUMA nodes (if enabled by GUC)
+ */
+ buffer_partitions_init();
+
/*
* Initialize all the buffer headers.
*/
@@ -144,6 +241,11 @@ BufferManagerShmemInit(void)
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
}
+ /*
+ * As this point we have all the buffers in a single long freelist. With
+ * freelist partitioning we rebuild them in StrategyInitialize.
+ */
+
/* Init other shared buffer-management stuff */
StrategyInitialize(!foundDescs);
@@ -152,24 +254,68 @@ BufferManagerShmemInit(void)
&backend_flush_after);
}
+/*
+ * Determine the size of memory page.
+ *
+ * XXX This is a bit tricky, because the result depends at which point we call
+ * this. Before the allocation we don't know if we succeed in allocating huge
+ * pages - but we have to size everything for the chance that we will. And then
+ * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
+ * pages. But at that point we can't adjust the sizing.
+ *
+ * XXX Maybe with huge_pages=try we should do the sizing twice - first with
+ * huge pages, and if that fails, then without them. But not for this patch.
+ * Up to this point there was no such dependency on huge pages.
+ */
+static Size
+get_memory_page_size(void)
+{
+ Size os_page_size;
+ Size huge_page_size;
+
+#ifdef WIN32
+ SYSTEM_INFO sysinfo;
+
+ GetSystemInfo(&sysinfo);
+ os_page_size = sysinfo.dwPageSize;
+#else
+ os_page_size = sysconf(_SC_PAGESIZE);
+#endif
+
+ /* assume huge pages get used, unless HUGE_PAGES_OFF */
+ if (huge_pages_status != HUGE_PAGES_OFF)
+ GetHugePageSize(&huge_page_size, NULL);
+ else
+ huge_page_size = 0;
+
+ return Max(os_page_size, huge_page_size);
+}
+
/*
* BufferManagerShmemSize
*
* compute the size of shared memory for the buffer pool including
* data pages, buffer descriptors, hash tables, etc.
+ *
+ * XXX Called before allocation, so we don't know if huge pages get used yet.
+ * So we need to assume huge pages get used, and use get_memory_page_size()
+ * to calculate the largest possible memory page.
*/
Size
BufferManagerShmemSize(void)
{
Size size = 0;
+ /* calculate partition info for buffers */
+ buffer_partitions_prepare();
+
/* size of buffer descriptors */
size = add_size(size, mul_size(NBuffers, sizeof(BufferDescPadded)));
/* to allow aligning buffer descriptors */
- size = add_size(size, PG_CACHE_LINE_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
/* size of data pages, plus alignment padding */
- size = add_size(size, PG_IO_ALIGN_SIZE);
+ size = add_size(size, Max(numa_page_size, PG_IO_ALIGN_SIZE));
size = add_size(size, mul_size(NBuffers, BLCKSZ));
/* size of stuff controlled by freelist.c */
@@ -184,5 +330,467 @@ BufferManagerShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* account for registry of NUMA partitions */
+ size = add_size(size, MAXALIGN(offsetof(BufferPartitions, partitions) +
+ mul_size(sizeof(BufferPartition), numa_partitions)));
+
return size;
}
+
+/*
+ * Calculate the NUMA node for a given buffer.
+ */
+int
+BufferGetNode(Buffer buffer)
+{
+ /* not NUMA interleaving */
+ if (numa_buffers_per_node == -1)
+ return 0;
+
+ return (buffer / numa_buffers_per_node);
+}
+
+/*
+ * pg_numa_interleave_memory
+ * move memory to different NUMA nodes in larger chunks
+ *
+ * startptr - start of the region (should be aligned to page size)
+ * endptr - end of the region (doesn't need to be aligned)
+ * mem_page_size - size of the memory page size
+ * chunk_size - size of the chunk to move to a single node (should be multiple
+ * of page size
+ * num_nodes - number of nodes to allocate memory to
+ *
+ * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
+ * That might be more efficient than numa_move_pages, as it works on larger
+ * chunks of memory, not individual system pages, I think.
+ *
+ * XXX The "interleave" name is not quite accurate, I guess.
+ */
+static void
+pg_numa_move_to_node(char *startptr, char *endptr, int node)
+{
+ Size mem_page_size;
+ Size sz;
+
+ /*
+ * Get the "actual" memory page size, not the one we used for sizing. We
+ * might have used huge page for sizing, but only get regular pages when
+ * allocating, so we must use the smaller pages here.
+ *
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ */
+ if (IsUnderPostmaster)
+ mem_page_size = pg_get_shmem_pagesize();
+ else
+ mem_page_size = get_memory_page_size();
+
+ Assert((int64) startptr % mem_page_size == 0);
+
+ sz = (endptr - startptr);
+ numa_tonode_memory(startptr, sz, node);
+}
+
+
+#define MIN_BUFFER_PARTITIONS 4
+
+/*
+ * buffer_partitions_prepare
+ * Calculate parameters for partitioning buffers.
+ *
+ * We want to split the shared buffers into multiple partitions, of roughly
+ * the same size. This is meant to serve multiple purposes. We want to map
+ * the partitions to different NUMA nodes, to balance memory usage, and
+ * allow partitioning some data structures built on top of buffers, to give
+ * preference to local access (buffers on the same NUMA node). This applies
+ * mostly to freelists and clocksweep.
+ *
+ * We may want to use partitioning even on non-NUMA systems, or when running
+ * on a single NUMA node. Partitioning the freelist/clocksweep is beneficial
+ * even without the NUMA effects.
+ *
+ * So we try to always build at least 4 partitions (MIN_BUFFER_PARTITIONS)
+ * in total, or at least one partition per NUMA node. We always create the
+ * same number of partitions per NUMA node.
+ *
+ * Some examples:
+ *
+ * - non-NUMA system (or 1 NUMA node): 4 partitions for the single node
+ *
+ * - 2 NUMA nodes: 4 partitions, 2 for each node
+ *
+ * - 3 NUMA nodes: 6 partitions, 2 for each node
+ *
+ * - 4+ NUMA nodes: one partition per node
+ *
+ * NUMA works on the memory-page granularity, which determines the smallest
+ * amount of memory we can allocate to single node. This is determined by
+ * how many BufferDescriptors fit onto a single memory page, so this depends
+ * on huge page support. With 2MB huge pages (typical on x86 Linux), this is
+ * 32768 buffers (256MB). With regular 4kB pages, it's 64 buffers (512KB).
+ *
+ * Note: This is determined before the allocation, i.e. we don't know if the
+ * allocation got to use huge pages. So unless huge_pages=off we assume we're
+ * using huge pages.
+ *
+ * This minimal size requirement only matters for the per-node amount of
+ * memory, not for the individual partitions. The partitions for the same
+ * node are a contiguous chunk of memory, which can be split arbitrarily,
+ * it's independent of the NUMA granularity.
+ *
+ * XXX This patch only implements placing the buffers onto different NUMA
+ * nodes. The freelist/clocksweep partitioning is implemented in separate
+ * patches later in the patch series. Those patches however use the same
+ * buffer partition registry, to align the partitions.
+ *
+ *
+ * XXX This needs to consider the minimum chunk size, i.e. we can't split
+ * buffers beyond some point, at some point it gets we run into the size of
+ * buffer descriptors. Not sure if we should give preference to one of these
+ * (probably at least print a warning).
+ *
+ * XXX We want to do this even with numa_buffers_interleave=false, so that the
+ * other patches can do their partitioning. But in that case we don't need to
+ * enforce the min chunk size (probably)?
+ *
+ * XXX We need to only call this once, when sizing the memory. But at that
+ * point we don't know if we get to use huge pages or not (unless when huge
+ * pages are disabled). We'll proceed as if the huge pages were used, and we
+ * may have to use larger partitions. Maybe there's some sort of fallback,
+ * but for now we simply disable the NUMA partitioning - it simply means the
+ * shared buffers are too small.
+ *
+ * XXX We don't need to make each partition a multiple of min_partition_size.
+ * That's something we need to do for a node (because NUMA works at granularity
+ * of pages), but partitions for a single node can split that arbitrarily.
+ * Although keeping the sizes power-of-two would allow calculating everything
+ * as shift/mask, without expensive division/modulo operations.
+ */
+static void
+buffer_partitions_prepare(void)
+{
+ /*
+ * Minimum number of buffers we can allocate to a NUMA node (determined by
+ * how many BufferDescriptors fit onto a memory page).
+ */
+ int min_node_buffers;
+
+ /*
+ * Maximum number of nodes we can split shared buffers to, assuming each
+ * node gets the smallest allocatable chunk (the last node can get a
+ * smaller amount of memory, not the full chunk).
+ */
+ int max_nodes;
+
+ /*
+ * How many partitions to create per node. Could be more than 1 for small
+ * number of nodes (of non-NUMA systems).
+ */
+ int num_partitions_per_node;
+
+ /* bail out if already initialized (calculate only once) */
+ if (numa_nodes != -1)
+ return;
+
+ /* XXX only gives us the number, the nodes may not be 0, 1, 2, ... */
+ numa_nodes = numa_num_configured_nodes();
+
+ /* XXX can this happen? */
+ if (numa_nodes < 1)
+ numa_nodes = 1;
+
+ elog(WARNING, "IsUnderPostmaster %d", IsUnderPostmaster);
+
+ /*
+ * XXX A bit weird. Do we need to worry about postmaster? Could this even
+ * run outside postmaster? I don't think so.
+ *
+ * XXX Another issue is we may get different values than when sizing the
+ * the memory, because at that point we didn't know if we get huge pages,
+ * so we assumed we will. Shouldn't cause crashes, but we might allocate
+ * shared memory and then not use some of it (because of the alignment
+ * that we don't actually need). Not sure about better way, good for now.
+ */
+ if (IsUnderPostmaster)
+ numa_page_size = pg_get_shmem_pagesize();
+ else
+ numa_page_size = get_memory_page_size();
+
+ /* make sure the chunks will align nicely */
+ Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
+ Assert(numa_page_size % sizeof(BufferDescPadded) == 0);
+ Assert(((BLCKSZ % numa_page_size) == 0) || ((numa_page_size % BLCKSZ) == 0));
+
+ /*
+ * The minimum number of buffers we can allocate from a single node, using
+ * the memory page size (determined by buffer descriptors). NUMA allocates
+ * memory in pages, and we need to do that for both buffers and
+ * descriptors at the same time.
+ *
+ * In practice the BLCKSZ doesn't really matter, because it's much larger
+ * than BufferDescPadded, so the result is determined buffer descriptors.
+ */
+ min_node_buffers = (numa_page_size / sizeof(BufferDescPadded));
+
+ /*
+ * Maximum number of nodes (each getting min_node_buffers) we can handle
+ * given the current shared buffers size. The last node is allowed to be
+ * smaller (half of the other nodes).
+ */
+ max_nodes = (NBuffers + (min_node_buffers / 2)) / min_node_buffers;
+
+ /*
+ * Can we actually do NUMA partitioning with these settings? If we can't
+ * handle the current number of nodes, then no.
+ *
+ * XXX This shouldn't be a big issue in practice. NUMA systems typically
+ * run with large shared buffers, which also makes the imbalance issues
+ * fairly significant (it's quick to rebalance 128MB, much slower to do
+ * that for 256GB).
+ */
+ numa_can_partition = true; /* assume we can allocate to nodes */
+ if (numa_nodes > max_nodes)
+ {
+ elog(WARNING, "shared buffers too small for %d nodes (max nodes %d)",
+ numa_nodes, max_nodes);
+ numa_can_partition = false;
+ }
+
+ /*
+ * We know we can partition to the desired number of nodes, now it's time
+ * to figure out how many partitions we need per node. We simply add
+ * partitions per node until we reach MIN_BUFFER_PARTITIONS.
+ *
+ * XXX Maybe we should make sure to keep the actual partition size a power
+ * of 2, to make the calculations simpler (shift instead of mod).
+ */
+ num_partitions_per_node = 1;
+
+ while (numa_nodes * num_partitions_per_node < MIN_BUFFER_PARTITIONS)
+ num_partitions_per_node++;
+
+ /* now we know the total number of partitions */
+ numa_partitions = (numa_nodes * num_partitions_per_node);
+
+ /*
+ * Finally, calculate how many buffers we'll assign to a single NUMA node.
+ * If we have only a single node, or can't map to that many nodes, just
+ * take a "fair share" of buffers.
+ *
+ * XXX In both cases the last node can get fewer buffers.
+ */
+ if (!numa_can_partition)
+ {
+ numa_buffers_per_node = (NBuffers + (numa_nodes - 1)) / numa_nodes;
+ }
+ else
+ {
+ numa_buffers_per_node = min_node_buffers;
+ while (numa_buffers_per_node * numa_nodes < NBuffers)
+ numa_buffers_per_node += min_node_buffers;
+
+ /* the last node should get at least some buffers */
+ Assert(NBuffers - (numa_nodes - 1) * numa_buffers_per_node > 0);
+ }
+
+ elog(LOG, "NUMA: buffers %d partitions %d num_nodes %d per_node %d buffers_per_node %d (min %d)",
+ NBuffers, numa_partitions, numa_nodes, num_partitions_per_node,
+ numa_buffers_per_node, min_node_buffers);
+}
+
+static void
+AssertCheckBufferPartitions(void)
+{
+#ifdef USE_ASSERT_CHECKING
+ int num_buffers = 0;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+
+ /*
+ * We can get a single-buffer partition, if the sizing forces the last
+ * partition to be just one buffer. But it's unlikely (and
+ * undesirable).
+ */
+ Assert(part->first_buffer <= part->last_buffer);
+ Assert((part->last_buffer - part->first_buffer + 1) == part->num_buffers);
+
+ num_buffers += part->num_buffers;
+
+ /*
+ * The first partition needs to start on buffer 0. Later partitions
+ * need to be contiguous, without skipping any buffers.
+ */
+ if (i == 0)
+ {
+ Assert(part->first_buffer == 0);
+ }
+ else
+ {
+ BufferPartition *prev = &BufferPartitionsArray->partitions[i - 1];
+
+ Assert((part->first_buffer - 1) == prev->last_buffer);
+ }
+
+ /* the last partition needs to end on buffer (NBuffers - 1) */
+ if (i == (numa_partitions - 1))
+ {
+ Assert(part->last_buffer == (NBuffers - 1));
+ }
+ }
+
+ Assert(num_buffers == NBuffers);
+#endif
+}
+
+static void
+buffer_partitions_init(void)
+{
+ int remaining_buffers = NBuffers;
+ int buffer = 0;
+ int parts_per_node = (numa_partitions / numa_nodes);
+ char *buffers_ptr,
+ *descriptors_ptr;
+
+ BufferPartitionsArray->npartitions = numa_partitions;
+
+ for (int n = 0; n < numa_nodes; n++)
+ {
+ /* buffers this node should get (last node can get fewer) */
+ int node_buffers = Min(remaining_buffers, numa_buffers_per_node);
+
+ /* split node buffers netween partitions (last one can get fewer) */
+ int part_buffers = (node_buffers + (parts_per_node - 1)) / parts_per_node;
+
+ remaining_buffers -= node_buffers;
+
+ Assert((node_buffers > 0) && (node_buffers <= NBuffers));
+ Assert((n >= 0) && (n < numa_nodes));
+
+ for (int p = 0; p < parts_per_node; p++)
+ {
+ int idx = (n * parts_per_node) + p;
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+ int num_buffers = Min(node_buffers, part_buffers);
+
+ Assert((idx >= 0) && (idx < numa_partitions));
+ Assert((buffer >= 0) && (buffer < NBuffers));
+ Assert((num_buffers > 0) && (num_buffers <= part_buffers));
+
+ /* XXX we should get the actual node ID from the mask */
+ part->numa_node = n;
+
+ part->num_buffers = num_buffers;
+ part->first_buffer = buffer;
+ part->last_buffer = buffer + (num_buffers - 1);
+
+ elog(LOG, "NUMA: buffer %d node %d partition %d buffers %d first %d last %d", idx, n, p, num_buffers, buffer, buffer + (num_buffers - 1));
+
+ buffer += num_buffers;
+ node_buffers -= part_buffers;
+ }
+ }
+
+ AssertCheckBufferPartitions();
+
+ /*
+ * With buffers interleaving disabled (or can't partition, because of
+ * shared buffers being too small), we're done.
+ */
+ if (!numa_buffers_interleave || !numa_can_partition)
+ return;
+
+ /*
+ * Assign chunks of buffers and buffer descriptors to the available NUMA
+ * nodes. We can't use the regular interleaving, because with regular
+ * memory pages (smaller than BLCKSZ) we'd split all buffers to multiple
+ * NUMA nodes. And we don't want that.
+ *
+ * But even with huge pages it seems like a good idea to not have mapping
+ * for each page.
+ *
+ * So we always assign a larger contiguous chunk of buffers to the same
+ * NUMA node, as calculated by choose_chunk_buffers(). We try to keep the
+ * chunks large enough to work both for buffers and buffer descriptors,
+ * but not too large. See the comments at choose_chunk_buffers() for
+ * details.
+ *
+ * Thanks to the earlier alignment (to memory page etc.), we know the
+ * buffers won't get split, etc.
+ *
+ * This also makes it easier / straightforward to calculate which NUMA
+ * node a buffer belongs to (it's a matter of divide + mod). See
+ * BufferGetNode().
+ *
+ * We need to account for partitions being of different length, when the
+ * NBuffers is not nicely divisible. To do that we keep track of the start
+ * of the next partition.
+ */
+ buffers_ptr = BufferBlocks;
+ descriptors_ptr = (char *) BufferDescriptors;
+
+ for (int i = 0; i < numa_partitions; i++)
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[i];
+ char *startptr,
+ *endptr;
+
+ /* first map buffers */
+ startptr = buffers_ptr;
+ endptr = startptr + ((Size) part->num_buffers * BLCKSZ);
+ buffers_ptr = endptr; /* start of the next partition */
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d buffers %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+
+ /* now do the same for buffer descriptors */
+ startptr = descriptors_ptr;
+ endptr = startptr + ((Size) part->num_buffers * sizeof(BufferDescPadded));
+ descriptors_ptr = endptr;
+
+ elog(LOG, "NUMA: buffer_partitions_init: %d => %d descriptors %d start %p end %p (size %ld)",
+ i, part->numa_node, part->num_buffers, startptr, endptr, (endptr - startptr));
+
+ pg_numa_move_to_node(startptr, endptr, part->numa_node);
+ }
+
+ /* we should have consumed the arrays exactly */
+ Assert(buffers_ptr == BufferBlocks + (Size) NBuffers * BLCKSZ);
+ Assert(descriptors_ptr == (char *) BufferDescriptors + (Size) NBuffers * sizeof(BufferDescPadded));
+}
+
+int
+BufferPartitionCount(void)
+{
+ return BufferPartitionsArray->npartitions;
+}
+
+void
+BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer)
+{
+ if ((idx >= 0) && (idx < BufferPartitionsArray->npartitions))
+ {
+ BufferPartition *part = &BufferPartitionsArray->partitions[idx];
+
+ *node = part->numa_node;
+ *num_buffers = part->num_buffers;
+ *first_buffer = part->first_buffer;
+ *last_buffer = part->last_buffer;
+
+ return;
+ }
+
+ elog(ERROR, "invalid partition index");
+}
+
+void
+BufferPartitionParams(int *num_partitions, int *num_nodes)
+{
+ *num_partitions = numa_partitions;
+ *num_nodes = numa_nodes;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..876cb64cf66 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -145,6 +145,9 @@ int max_worker_processes = 8;
int max_parallel_workers = 8;
int MaxBackends = 0;
+/* NUMA stuff */
+bool numa_buffers_interleave = false;
+
/* GUC parameters for vacuum */
int VacuumBufferUsageLimit = 2048;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d14b1678e7f..9570087aa60 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2116,6 +2116,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"numa_buffers_interleave", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Enables NUMA interleaving of shared buffers."),
+ gettext_noop("When enabled, the buffers in shared memory are interleaved to all NUMA nodes."),
+ },
+ &numa_buffers_interleave,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY,
gettext_noop("Enables a physical standby to synchronize logical failover replication slots from the primary server."),
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..014a6079af2 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT bool numa_buffers_interleave;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 52a71b138f7..9dfbecb9fe4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -323,6 +323,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
+extern PGDLLIMPORT BufferPartitions *BufferPartitionsArray;
extern PGDLLIMPORT ConditionVariableMinimallyPadded *BufferIOCVArray;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
@@ -491,4 +492,9 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+extern int BufferPartitionCount(void);
+extern void BufferPartitionGet(int idx, int *node, int *num_buffers,
+ int *first_buffer, int *last_buffer);
+extern void BufferPartitionParams(int *num_partitions, int *num_nodes);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 41fdc1e7693..deaf4f19fa4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -143,6 +143,20 @@ struct ReadBuffersOperation
typedef struct ReadBuffersOperation ReadBuffersOperation;
+typedef struct BufferPartition
+{
+ int numa_node;
+ int num_buffers;
+ int first_buffer;
+ int last_buffer;
+} BufferPartition;
+
+typedef struct BufferPartitions
+{
+ int npartitions;
+ BufferPartition partitions[FLEXIBLE_ARRAY_MEMBER];
+} BufferPartitions;
+
/* forward declared, to avoid having to expose buf_internals.h here */
struct WritebackContext;
@@ -319,6 +333,7 @@ extern void EvictRelUnpinnedBuffers(Relation rel,
/* in buf_init.c */
extern void BufferManagerShmemInit(void);
extern Size BufferManagerShmemSize(void);
+extern int BufferGetNode(Buffer buffer);
/* in localbuf.c */
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3daba26b237..c695cfa76e8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -346,6 +346,8 @@ BufferDescPadded
BufferHeapTupleTableSlot
BufferLookupEnt
BufferManagerRelation
+BufferPartition
+BufferPartitions
BufferStrategyControl
BufferTag
BufferUsage
--
2.50.1
[text/csv] numa-hb176.csv (17.8K, ../[email protected]/9-numa-hb176.csv)
download | inline:
shared_buffers mode build pinning tps tps_25 tps_50 tps_75 tps_95 tps_99 lat_avg lat_stddev sb_warmup sb_benchmark
32GB simple numa-0-master none 2094014.794767 2073550.4 2082788.6 2102633.4 2167920.9 2184485.4 0.061 0.020 100.00 100.00
32GB simple numa-0-master random 1286271.933424 1284754.0 1286494.7 1288674.1 1291197.6 1292495.7 0.099 0.036 100.00 100.00
32GB simple numa-0-master colocated 2388811.232587 2383281.7 2389972.1 2396598.3 2401910.3 2404253.9 0.053 0.012 100.00 100.00
32GB simple numa-1-buffers none 2047019.576932 2033460.0 2042257.4 2055448.4 2085925.9 2114769.4 0.062 0.024 100.00 100.00
32GB simple numa-1-buffers random 1192511.104640 1191714.2 1192982.1 1194393.7 1196299.0 1197481.5 0.107 0.035 100.00 100.00
32GB simple numa-1-buffers colocated 2405932.896675 2398925.4 2407886.6 2415869.6 2420222.4 2422280.8 0.053 0.012 100.00 100.00
32GB simple numa-2-localalloc none 2069848.559853 2055515.7 2066642.7 2079182.0 2115517.3 2130680.9 0.062 0.028 100.00 100.00
32GB simple numa-2-localalloc random 1206669.031819 1205724.8 1207054.5 1208558.7 1210337.3 1211225.3 0.106 0.034 100.00 100.00
32GB simple numa-2-localalloc colocated 2357590.782369 2354531.3 2359009.7 2363536.6 2367643.1 2368925.9 0.054 0.012 100.00 100.00
32GB simple numa-3-no-tail none 2076592.800331 2056606.8 2069066.2 2087446.7 2140462.9 2159359.8 0.061 0.025 100.00 100.00
32GB simple numa-3-no-tail random 1201742.073521 1223346.1 1224697.3 1226182.0 1227864.0 1228923.4 0.106 0.038 100.00 100.00
32GB simple numa-3-no-tail colocated 2404952.281461 2397528.5 2406571.7 2414327.2 2419760.7 2422838.3 0.053 0.012 100.00 100.00
32GB simple numa-4-freelist none 2144928.827252 2127996.5 2140287.4 2156668.7 2200868.4 2215117.6 0.060 0.049 100.00 100.00
32GB simple numa-4-freelist random 1262573.059680 1262244.5 1263434.2 1264747.4 1266127.9 1267152.6 0.101 0.054 100.00 100.00
32GB simple numa-4-freelist colocated 2452015.746479 2445803.4 2453721.3 2461334.6 2465615.6 2466924.9 0.052 0.042 100.00 100.00
32GB simple numa-5-clocksweep none 2147727.945415 2130889.9 2146203.3 2162137.9 2189644.5 2227659.6 0.059 0.035 25.00 100.00
32GB simple numa-5-clocksweep random 1241338.787551 1241416.4 1242391.2 1243166.6 1243964.3 1244688.1 0.103 0.045 25.00 100.00
32GB simple numa-5-clocksweep colocated 2545190.769146 2545681.2 2546922.8 2547753.0 2548595.1 2549195.3 0.050 0.028 25.00 100.00
32GB simple numa-6-pgproc none 2180410.964299 2157924.5 2170931.6 2192210.9 2249331.0 2272390.9 0.059 0.045 25.00 100.00
32GB simple numa-6-pgproc random 1240799.452682 1240842.3 1241607.5 1242474.2 1243488.1 1243901.1 0.103 0.045 25.00 100.00
32GB simple numa-6-pgproc colocated 2636934.271698 2637377.9 2638581.9 2639457.1 2641224.1 2642282.5 0.048 0.023 25.00 100.00
32GB simple numa-7-pinning none 2101452.917588 2090545.5 2103858.4 2118982.5 2130923.9 2135597.8 0.061 0.030 25.00 100.00
32GB simple numa-7-pinning random 1238919.511046 1238852.6 1239708.4 1240412.1 1241329.5 1241757.2 0.103 0.046 25.00 100.00
32GB simple numa-7-pinning colocated 2612797.027980 2613302.4 2614308.2 2615035.3 2616057.0 2616497.2 0.049 0.031 25.00 100.00
32GB prepared numa-0-master none 2600265.035719 2594175.1 2603657.8 2614298.2 2627655.9 2639079.8 0.049 0.032 100.00 100.00
32GB prepared numa-0-master random 609278.573839 601973.9 608734.8 615687.7 626392.1 643183.4 0.209 0.082 100.00 100.00
32GB prepared numa-0-master colocated 2936188.002438 2931071.7 2937149.1 2942793.5 2951091.2 2955221.7 0.043 0.015 100.00 100.00
32GB prepared numa-1-buffers none 2556483.026566 2547839.7 2558098.4 2568563.8 2582199.4 2597160.4 0.050 0.034 100.00 100.00
32GB prepared numa-1-buffers random 643264.098458 638733.7 645325.8 651338.1 657511.7 659828.3 0.198 0.086 100.00 100.00
32GB prepared numa-1-buffers colocated 2896245.824802 2885755.9 2896842.2 2906640.6 2918054.9 2922427.6 0.044 0.014 100.00 100.00
32GB prepared numa-2-localalloc none 2595430.169568 2584462.8 2597788.8 2609027.1 2623358.2 2635969.4 0.049 0.034 100.00 100.00
32GB prepared numa-2-localalloc random 594695.123377 589296.0 593619.3 600417.6 609539.2 612091.0 0.214 0.081 100.00 100.00
32GB prepared numa-2-localalloc colocated 2918297.926587 2909171.3 2918484.3 2927639.3 2938536.1 2943213.7 0.044 0.014 100.00 100.00
32GB prepared numa-3-no-tail none 2572010.525302 2561524.7 2574921.5 2586000.7 2604077.8 2617103.1 0.050 0.034 100.00 100.00
32GB prepared numa-3-no-tail random 607225.735077 598681.9 604986.4 615368.3 627310.5 634224.9 0.210 0.080 100.00 100.00
32GB prepared numa-3-no-tail colocated 2903377.427631 2899103.0 2904451.2 2908960.5 2914982.7 2919139.5 0.044 0.014 100.00 100.00
32GB prepared numa-4-freelist none 2656080.200019 2647042.2 2657972.3 2671358.7 2694405.7 2707741.6 0.048 0.042 100.00 100.00
32GB prepared numa-4-freelist random 1487277.158053 1492190.7 1493668.9 1495124.8 1496979.6 1498150.4 0.086 0.041 100.00 100.00
32GB prepared numa-4-freelist colocated 3033967.176021 3029920.1 3034382.4 3038608.2 3045720.7 3047922.2 0.042 0.022 100.00 100.00
32GB prepared numa-5-clocksweep none 2832317.609128 2823434.5 2836798.6 2851170.0 2872311.7 2885272.1 0.045 0.041 25.00 100.00
32GB prepared numa-5-clocksweep random 1461263.713983 1480257.7 1481681.0 1482839.0 1484390.0 1485436.7 0.087 0.044 25.00 100.00
32GB prepared numa-5-clocksweep colocated 3287319.038876 3287489.6 3288874.0 3289849.3 3291265.6 3291769.0 0.039 0.024 25.00 100.00
32GB prepared numa-6-pgproc none 2864731.663543 2856449.9 2868483.2 2880807.6 2897585.8 2912240.6 0.045 0.040 25.00 100.00
32GB prepared numa-6-pgproc random 1490324.471864 1507399.4 1509042.0 1510167.1 1511607.1 1512726.9 0.086 0.044 25.00 100.00
32GB prepared numa-6-pgproc colocated 3332521.391978 3333023.7 3333983.0 3334822.0 3336166.3 3336745.6 0.038 0.028 25.00 100.00
32GB prepared numa-7-pinning none 3196743.972868 3195967.9 3204768.7 3214707.2 3228582.4 3235446.2 0.040 0.032 25.00 100.00
32GB prepared numa-7-pinning random 1497592.952405 1515042.8 1516513.1 1517717.4 1519105.0 1519837.0 0.085 0.043 25.00 100.00
32GB prepared numa-7-pinning colocated 3366788.066030 3364820.9 3366270.6 3371283.5 3374229.5 3375696.5 0.038 0.023 25.00 100.00
16GB simple numa-0-master none 1827078.467315 1817756.3 1825443.1 1832543.8 1859686.9 1870789.3 0.070 0.024 100.00 100.00
16GB simple numa-0-master random 1204011.746569 1203116.3 1204501.7 1205861.8 1207978.8 1208888.9 0.106 0.051 100.00 100.00
16GB simple numa-0-master colocated 2091429.835432 2091732.8 2092492.3 2093089.4 2094002.7 2094487.2 0.061 0.016 100.00 100.00
16GB simple numa-1-buffers none 1818607.320519 1810580.9 1817367.6 1825469.1 1841348.0 1852768.7 0.070 0.029 100.00 100.00
16GB simple numa-1-buffers random 1137821.791547 1136829.3 1138436.1 1139733.2 1141115.9 1142075.5 0.112 0.043 100.00 100.00
16GB simple numa-1-buffers colocated 2070456.921423 2070408.5 2071031.2 2071629.6 2072642.0 2073077.9 0.062 0.015 100.00 100.00
16GB simple numa-2-localalloc none 1814337.964368 1802150.1 1809157.6 1818835.4 1859212.0 1879167.6 0.070 0.030 100.00 100.00
16GB simple numa-2-localalloc random 1140136.209840 1139137.5 1140471.3 1141764.2 1143336.5 1144688.4 0.112 0.045 100.00 100.00
16GB simple numa-2-localalloc colocated 2046749.245045 2046747.3 2047343.3 2047936.5 2048622.9 2049092.8 0.062 0.015 100.00 100.00
16GB simple numa-3-no-tail none 1788677.591511 1779668.7 1786752.4 1794861.3 1816346.7 1834453.2 0.071 0.037 100.00 100.00
16GB simple numa-3-no-tail random 1147815.761542 1146738.5 1148289.5 1149497.5 1151206.6 1152309.6 0.111 0.044 100.00 100.00
16GB simple numa-3-no-tail colocated 2063855.159562 2063587.2 2064405.6 2065153.3 2065818.4 2066994.5 0.062 0.015 100.00 100.00
16GB simple numa-4-freelist none 1872879.183670 1861914.8 1869084.4 1878799.0 1913317.6 1933159.1 0.068 0.033 100.00 100.00
16GB simple numa-4-freelist random 1207729.009946 1206622.2 1208305.2 1209726.6 1211618.5 1213711.5 0.106 0.053 100.00 100.00
16GB simple numa-4-freelist colocated 2135628.758077 2135447.6 2136058.9 2136774.1 2137744.9 2138242.8 0.060 0.026 100.00 100.00
16GB simple numa-5-clocksweep none 2016983.483955 2002764.2 2014062.2 2026487.8 2060396.1 2069378.2 0.063 0.034 25.00 100.00
16GB simple numa-5-clocksweep random 1215517.698425 1214417.6 1215885.8 1217436.2 1219034.0 1220391.1 0.105 0.042 25.00 100.00
16GB simple numa-5-clocksweep colocated 2365377.989509 2365342.3 2365867.5 2366356.9 2367174.3 2367813.4 0.054 0.015 25.00 100.00
16GB simple numa-6-pgproc none 1988517.293447 1976851.9 1985550.0 1997500.1 2022116.5 2040373.8 0.064 0.038 25.00 100.00
16GB simple numa-6-pgproc random 1179440.500877 1178373.3 1179920.9 1181132.7 1182906.5 1184106.3 0.108 0.048 25.01 100.00
16GB simple numa-6-pgproc colocated 2345242.334786 2345206.5 2345775.2 2346357.8 2347204.3 2347987.4 0.054 0.017 25.01 100.00
16GB simple numa-7-pinning none 2143131.371367 2137376.4 2145502.7 2153375.1 2160641.8 2163537.3 0.060 0.028 25.01 100.00
16GB simple numa-7-pinning random 1188670.161623 1187596.6 1189122.9 1190417.1 1192084.0 1193320.3 0.108 0.045 25.00 100.00
16GB simple numa-7-pinning colocated 2403120.900906 2402640.4 2403340.4 2404187.0 2406320.0 2407336.8 0.053 0.016 25.01 100.00
16GB prepared numa-0-master none 2221009.728590 2217463.4 2226291.2 2234013.4 2245377.8 2256120.3 0.057 0.041 100.00 100.00
16GB prepared numa-0-master random 602510.378457 593188.6 603924.9 610132.7 619242.5 633291.3 0.212 0.091 100.00 100.00
16GB prepared numa-0-master colocated 2481978.582055 2481482.9 2482651.3 2483698.2 2484969.3 2485873.2 0.051 0.018 100.00 100.00
16GB prepared numa-1-buffers none 2193555.966068 2189913.6 2198171.8 2207163.1 2218853.3 2229229.3 0.058 0.043 100.00 100.00
16GB prepared numa-1-buffers random 613141.518268 603012.6 611405.3 622477.1 633218.1 648076.5 0.208 0.078 100.00 100.00
16GB prepared numa-1-buffers colocated 2496646.168791 2496647.0 2497740.0 2498703.0 2499768.5 2500156.5 0.051 0.017 100.00 100.00
16GB prepared numa-2-localalloc none 2213277.504474 2208962.0 2218383.2 2227701.0 2239536.7 2251144.5 0.058 0.043 100.00 100.00
16GB prepared numa-2-localalloc random 618355.985374 611937.5 618198.9 623245.7 636857.7 644933.8 0.207 0.081 100.00 100.00
16GB prepared numa-2-localalloc colocated 2502981.656459 2502711.5 2503524.0 2504358.4 2505381.9 2506685.0 0.051 0.017 100.00 100.00
16GB prepared numa-3-no-tail none 2211849.394014 2206713.7 2216419.3 2225485.4 2237445.0 2248063.5 0.058 0.043 100.00 100.00
16GB prepared numa-3-no-tail random 575000.639904 570502.7 576741.1 580100.8 585510.7 590986.2 0.222 0.096 100.00 100.00
16GB prepared numa-3-no-tail colocated 2460474.838856 2460029.2 2461049.4 2462003.9 2463196.9 2463833.2 0.052 0.018 100.00 100.00
16GB prepared numa-4-freelist none 2281726.084202 2277193.2 2288404.0 2298464.9 2310718.6 2319249.1 0.056 0.045 100.00 100.00
16GB prepared numa-4-freelist random 1398663.617786 1398797.0 1400908.7 1403105.2 1406152.4 1409074.7 0.091 0.052 100.00 100.00
16GB prepared numa-4-freelist colocated 2679995.292637 2679985.9 2680862.0 2681547.4 2682665.8 2683429.4 0.048 0.028 100.00 100.00
16GB prepared numa-5-clocksweep none 2540095.692745 2532340.2 2545940.3 2560762.3 2577130.8 2589321.1 0.050 0.046 25.01 100.00
16GB prepared numa-5-clocksweep random 1453707.189064 1459045.1 1461321.8 1463586.2 1466581.8 1467714.9 0.088 0.050 25.00 100.00
16GB prepared numa-5-clocksweep colocated 2868908.990574 2868459.3 2869305.9 2869931.4 2870962.4 2872039.0 0.045 0.019 25.01 100.00
16GB prepared numa-6-pgproc none 2514531.754697 2508478.5 2520232.0 2532724.4 2555334.2 2572128.4 0.051 0.051 25.00 100.00
16GB prepared numa-6-pgproc random 1421942.962229 1425471.6 1427251.5 1429185.0 1432119.4 1434213.6 0.090 0.048 25.01 100.00
16GB prepared numa-6-pgproc colocated 2895460.904497 2894977.4 2895783.9 2896473.0 2897331.0 2897774.0 0.044 0.020 25.01 100.00
16GB prepared numa-7-pinning none 2570293.162052 2566575.5 2577298.2 2586033.6 2597683.7 2605701.0 0.050 0.045 25.01 100.00
16GB prepared numa-7-pinning random 1419469.356241 1422140.4 1424287.1 1425895.0 1429015.0 1431937.3 0.090 0.048 25.00 100.00
16GB prepared numa-7-pinning colocated 2965399.304227 2964597.4 2965842.7 2966982.3 2968241.0 2969082.9 0.043 0.018 25.00 100.00
48GB simple numa-0-master none 2240626.513398 2217063.6 2239195.3 2265723.7 2301549.5 2341012.7 0.057 0.118 83.07 100.00
48GB simple numa-0-master random 1318468.781539 1319007.2 1320126.3 1321273.5 1322828.4 1323737.2 0.097 0.162 84.09 100.00
48GB simple numa-0-master colocated 2699394.144782 2688747.8 2697168.5 2716980.8 2731581.3 2733854.0 0.047 0.108 82.93 100.00
48GB simple numa-1-buffers none 2245154.802833 2226344.9 2240563.6 2260432.1 2315478.3 2349275.6 0.057 0.122 83.31 100.00
48GB simple numa-1-buffers random 1261172.614367 1262162.4 1262880.0 1263940.7 1265028.8 1265850.8 0.101 0.164 83.59 100.00
48GB simple numa-1-buffers colocated 2650623.262153 2642051.6 2648415.7 2668555.3 2680209.3 2683861.3 0.048 0.108 81.91 100.00
48GB simple numa-2-localalloc none 2249367.782141 2228686.8 2248606.8 2273746.4 2312259.7 2331650.3 0.057 0.136 82.87 100.00
48GB simple numa-2-localalloc random 1244674.409409 1245027.6 1246358.7 1247559.9 1248836.7 1249097.2 0.103 0.164 82.72 100.00
48GB simple numa-2-localalloc colocated 2611193.021413 2600647.0 2609337.8 2628624.3 2640684.6 2643435.3 0.049 0.109 82.64 100.00
48GB simple numa-3-no-tail none 2268008.571187 2240616.6 2261884.9 2296781.3 2348528.0 2368243.7 0.056 0.119 82.66 100.00
48GB simple numa-3-no-tail random 1233026.884272 1233540.6 1234897.1 1236002.0 1237262.7 1237872.8 0.104 0.159 82.54 100.00
48GB simple numa-3-no-tail colocated 2658156.684861 2651293.0 2656574.6 2673886.2 2682779.0 2684405.4 0.048 0.114 82.15 100.00
48GB simple numa-4-freelist none 2251576.603050 2230542.2 2244920.3 2270473.5 2337061.4 2352738.5 0.057 0.150 83.02 100.00
48GB simple numa-4-freelist random 1307933.407227 1308092.8 1309804.3 1311297.6 1312511.3 1313296.8 0.098 0.135 82.57 100.00
48GB simple numa-4-freelist colocated 2601537.595536 2584555.7 2598362.4 2622709.5 2638031.8 2640277.4 0.049 0.072 81.97 100.00
48GB simple numa-5-clocksweep none 2241382.241155 2218548.3 2237248.1 2265353.8 2295880.7 2323149.0 0.057 0.038 25.00 100.00
48GB simple numa-5-clocksweep random 1262514.317310 1263358.9 1264072.0 1264592.4 1265358.6 1265534.2 0.101 0.053 25.00 100.00
48GB simple numa-5-clocksweep colocated 2610661.390468 2611730.6 2612727.7 2613709.4 2614884.8 2615970.1 0.049 0.030 25.00 100.00
48GB simple numa-6-pgproc none 2336305.373280 2301276.0 2324107.3 2360639.4 2436318.1 2466602.9 0.055 0.037 25.00 100.00
48GB simple numa-6-pgproc random 1273021.136908 1273754.9 1274463.0 1275048.7 1275733.1 1275980.7 0.100 0.054 25.00 100.00
48GB simple numa-6-pgproc colocated 2774107.243898 2775455.7 2776917.8 2777976.2 2779298.2 2780131.0 0.046 0.032 25.00 100.00
48GB simple numa-7-pinning none 2415318.045028 2407691.0 2422663.2 2433578.9 2450156.5 2467841.0 0.053 0.043 25.00 100.00
48GB simple numa-7-pinning random 1252468.481863 1253129.0 1253795.5 1254362.7 1254986.2 1255774.8 0.102 0.054 25.00 100.00
48GB simple numa-7-pinning colocated 2780002.945371 2781441.5 2783055.8 2784023.5 2785046.9 2785595.3 0.046 0.029 25.00 100.00
48GB prepared numa-0-master none 2923496.551126 2905774.4 2922465.4 2939254.6 2963572.9 2984084.0 0.044 0.029 100.00 100.00
48GB prepared numa-0-master random 587603.235381 582356.7 586515.2 591790.0 598064.6 627649.2 0.218 0.087 100.00 100.00
48GB prepared numa-0-master colocated 3299794.118436 3287550.0 3294055.9 3306638.1 3332989.3 3335413.0 0.039 0.011 100.00 100.00
48GB prepared numa-1-buffers none 2831238.524941 2814327.5 2830103.8 2847890.1 2873928.6 2888011.2 0.045 0.030 100.00 100.00
48GB prepared numa-1-buffers random 609794.507428 602144.0 607516.3 618335.2 626322.2 632261.4 0.210 0.084 100.00 100.00
48GB prepared numa-1-buffers colocated 3231774.005677 3218329.5 3225551.4 3244716.2 3263817.8 3269024.8 0.039 0.011 100.00 100.00
48GB prepared numa-2-localalloc none 2890615.623142 2871459.0 2889893.6 2906771.8 2928779.7 2949137.3 0.044 0.029 100.00 100.00
48GB prepared numa-2-localalloc random 597263.592382 589548.2 595436.5 602463.6 614818.7 624720.2 0.214 0.085 100.00 100.00
48GB prepared numa-2-localalloc colocated 3313277.527719 3298239.7 3307092.5 3327513.0 3345337.3 3351738.9 0.039 0.011 100.00 100.00
48GB prepared numa-3-no-tail none 2880257.694598 2860167.2 2877982.3 2895775.0 2925540.9 2942045.3 0.044 0.028 100.00 100.00
48GB prepared numa-3-no-tail random 598932.466701 593835.6 599657.4 604247.1 610151.9 613496.6 0.213 0.084 100.00 100.00
48GB prepared numa-3-no-tail colocated 3242629.469947 3230052.0 3236094.3 3254957.5 3274484.5 3279944.4 0.039 0.011 100.00 100.00
48GB prepared numa-4-freelist none 2923127.852195 2908968.9 2926129.9 2947127.7 2979743.3 2997703.0 0.044 0.157 100.00 100.00
48GB prepared numa-4-freelist random 1513129.413478 1515095.4 1516737.2 1518212.7 1519615.5 1520348.8 0.084 0.152 100.00 100.00
48GB prepared numa-4-freelist colocated 3395256.901017 3387587.5 3390973.7 3406014.7 3425695.7 3431188.9 0.038 0.070 100.00 100.00
48GB prepared numa-5-clocksweep none 2965437.145038 2949924.3 2965560.5 2985599.7 3008788.0 3020367.0 0.043 0.041 25.00 100.00
48GB prepared numa-5-clocksweep random 632449.314565 603792.6 613986.6 638880.6 758179.3 824435.2 0.202 0.099 25.00 100.00
48GB prepared numa-5-clocksweep colocated 3476865.561664 3477458.2 3478930.4 3480293.6 3482151.3 3483126.4 0.037 0.028 25.00 100.00
48GB prepared numa-6-pgproc none 3005569.927865 2989976.9 3007566.8 3023434.9 3052049.5 3073108.0 0.042 0.041 25.00 100.00
48GB prepared numa-6-pgproc random 607680.562253 579408.9 584803.8 600815.2 689898.4 975531.4 0.210 0.104 25.00 100.00
48GB prepared numa-6-pgproc colocated 3486467.905993 3487855.4 3489198.7 3490324.1 3492199.3 3492894.6 0.037 0.035 25.00 100.00
48GB prepared numa-7-pinning none 3177862.124437 3171786.2 3185876.7 3201378.0 3214490.6 3216137.0 0.040 0.045 25.00 100.00
48GB prepared numa-7-pinning random 626214.083568 602386.2 610697.5 619334.3 742241.2 906615.1 0.204 0.095 25.00 100.00
48GB prepared numa-7-pinning colocated 3562390.486753 3563016.5 3564196.8 3565135.0 3566639.3 3568153.2 0.036 0.029 25.00 100.00
[application/x-shellscript] run-huge-pages.sh (4.6K, ../[email protected]/10-run-huge-pages.sh)
download
[text/csv] numa-xeon.csv (17.0K, ../[email protected]/11-numa-xeon.csv)
download | inline:
shared_buffers mode build tps tps_25 tps_50 tps_75 tps_95 tps_99 lat_avg lat_stddev sb_warmup sb_benchmark
16GB simple numa-0-master none 435660.835849 436569.7 437207.0 437769.1 438452.7 439080.9 0.101 0.011 100.00 100.00
16GB simple numa-0-master random 435079.457369 437119.3 437506.1 437756.3 438134.0 438291.5 0.101 0.024 100.00 100.00
16GB simple numa-0-master colocated 436358.722452 437406.6 437615.3 437874.0 438046.3 438115.3 0.100 0.013 100.00 100.00
16GB simple numa-1-buffers none 439427.372422 440365.9 440997.7 441622.7 442693.4 443225.0 0.100 0.010 100.00 100.00
16GB simple numa-1-buffers random 435027.840470 437371.8 437548.6 437674.4 437835.1 437977.1 0.101 0.024 100.00 100.00
16GB simple numa-1-buffers colocated 450595.291038 451412.6 451896.2 452269.9 452706.8 452926.4 0.097 0.014 100.00 100.00
16GB simple numa-2-localalloc none 436134.188747 437118.8 437674.4 438109.5 439128.5 439655.1 0.101 0.011 100.00 100.00
16GB simple numa-2-localalloc random 437572.460095 439665.5 440029.2 440257.7 440517.3 440679.6 0.100 0.023 100.00 100.00
16GB simple numa-2-localalloc colocated 444300.471660 445406.9 445618.0 445825.0 446094.6 446246.4 0.099 0.014 100.00 100.00
16GB simple numa-3-no-tail none 442966.018997 443868.4 444521.3 445183.2 446048.3 446656.7 0.099 0.023 100.00 100.00
16GB simple numa-3-no-tail random 424481.739349 426572.7 426769.8 426986.1 427158.8 427255.8 0.103 0.024 100.00 100.00
16GB simple numa-3-no-tail colocated 446510.670808 447743.0 447996.2 448224.2 448483.6 448665.8 0.098 0.014 100.00 100.00
16GB simple numa-4-freelist none 434221.725822 435234.3 435867.1 436456.9 437180.8 437682.7 0.101 0.024 100.00 100.00
16GB simple numa-4-freelist random 433152.504959 435534.9 435810.3 436072.8 436416.5 436578.0 0.101 0.028 100.00 100.00
16GB simple numa-4-freelist colocated 455622.595281 456832.0 457102.8 457341.8 457665.1 457882.3 0.096 0.019 100.00 100.00
16GB simple numa-5-clocksweep none 440814.327529 441999.8 442477.1 443091.1 443836.2 444687.0 0.099 0.015 50.00 100.00
16GB simple numa-5-clocksweep random 432098.974783 434509.3 434730.0 434871.7 435060.8 435180.0 0.102 0.027 25.01 100.00
16GB simple numa-5-clocksweep colocated 447123.967715 448425.4 448583.4 448774.8 448957.4 449042.6 0.098 0.018 50.00 100.00
16GB simple numa-6-pgproc none 444542.151597 445826.3 446356.1 446856.9 447681.5 448287.1 0.099 0.017 25.01 100.00
16GB simple numa-6-pgproc random 437828.906615 440373.6 440689.5 440846.6 441058.2 441178.0 0.100 0.028 25.01 100.00
16GB simple numa-6-pgproc colocated 455042.538054 456570.6 456712.2 456849.8 457011.3 457107.5 0.096 0.017 50.01 100.00
16GB simple numa-7-pinning none 441672.926416 442804.5 443432.7 444105.0 445128.0 446039.1 0.099 0.017 26.86 100.00
16GB simple numa-7-pinning random 426223.045224 428477.9 428677.4 428851.0 429036.2 429105.0 0.103 0.026 50.01 100.00
16GB simple numa-7-pinning colocated 450392.113450 451822.6 451924.7 452028.9 452180.4 452238.1 0.097 0.017 50.01 100.00
16GB prepared numa-0-master none 780270.504321 781452.2 783365.3 784805.8 787342.7 788523.2 0.056 0.008 100.00 100.00
16GB prepared numa-0-master random 740319.460261 744682.7 745343.8 745970.8 746679.9 747127.0 0.059 0.022 100.00 100.00
16GB prepared numa-0-master colocated 808745.174879 810974.9 811570.6 812054.2 812866.5 813370.0 0.054 0.010 100.00 100.00
16GB prepared numa-1-buffers none 779061.549666 780355.0 782098.7 783859.4 786243.6 787488.0 0.056 0.008 100.00 100.00
16GB prepared numa-1-buffers random 731931.525311 735979.8 736774.3 737560.1 738127.5 738339.2 0.060 0.022 100.00 100.00
16GB prepared numa-1-buffers colocated 834043.848216 835960.7 836704.2 837765.5 838531.0 838780.4 0.052 0.010 100.00 100.00
16GB prepared numa-2-localalloc none 776686.423107 778070.3 779738.9 781332.7 783734.7 784749.5 0.056 0.008 100.00 100.00
16GB prepared numa-2-localalloc random 746071.021032 750606.0 751054.7 751520.2 752190.4 752477.2 0.059 0.022 100.00 100.00
16GB prepared numa-2-localalloc colocated 845194.301526 847919.8 848270.9 848585.7 849056.6 849324.5 0.052 0.011 100.00 100.00
16GB prepared numa-3-no-tail none 767472.661364 768654.5 770426.5 772322.9 774160.3 775354.7 0.057 0.008 100.00 100.00
16GB prepared numa-3-no-tail random 735100.627273 739765.5 740399.9 740837.1 741499.4 741808.5 0.060 0.022 100.00 100.00
16GB prepared numa-3-no-tail colocated 824897.098984 827288.2 827852.2 828430.8 828920.2 829226.3 0.053 0.010 100.00 100.00
16GB prepared numa-4-freelist none 759776.605571 760861.3 762909.4 764281.0 766359.8 768521.9 0.058 0.014 100.00 100.00
16GB prepared numa-4-freelist random 729800.689065 734337.4 734947.1 735428.8 736218.4 736975.4 0.060 0.041 100.00 100.00
16GB prepared numa-4-freelist colocated 846908.222612 849617.2 850197.0 850741.3 851328.2 851774.7 0.052 0.019 100.00 100.00
16GB prepared numa-5-clocksweep none 783444.253474 785122.5 786840.3 788354.2 790602.0 792040.7 0.056 0.012 50.01 100.00
16GB prepared numa-5-clocksweep random 738274.074795 742770.6 743492.0 743935.3 744471.7 744799.4 0.059 0.023 50.00 100.00
16GB prepared numa-5-clocksweep colocated 874129.726758 877734.8 878238.5 878509.2 878937.3 879362.2 0.050 0.013 50.01 100.00
16GB prepared numa-6-pgproc none 776320.601272 777672.8 779517.1 781352.5 783360.7 786641.0 0.056 0.012 50.00 100.00
16GB prepared numa-6-pgproc random 748473.862569 752733.0 753614.7 754762.8 755471.1 755954.3 0.059 0.024 50.00 100.00
16GB prepared numa-6-pgproc colocated 858900.054256 861867.8 862316.6 862654.1 863077.5 863358.5 0.051 0.013 50.01 100.00
16GB prepared numa-7-pinning none 778286.188920 780018.7 781462.8 782675.8 784586.6 786637.1 0.056 0.014 25.01 100.00
16GB prepared numa-7-pinning random 746250.393130 751720.5 752065.0 752399.7 752909.5 753307.0 0.059 0.032 25.01 100.00
16GB prepared numa-7-pinning colocated 860645.977827 863759.1 864312.1 864970.8 865869.3 866131.5 0.051 0.014 25.01 100.00
8GB simple numa-0-master none 420481.702806 421397.6 421991.9 422476.7 423331.7 423793.6 0.104 0.012 100.00 100.00
8GB simple numa-0-master random 406979.498757 408972.2 409116.6 409268.8 409457.2 409527.0 0.108 0.026 100.00 100.00
8GB simple numa-0-master colocated 421026.362476 421910.1 422224.4 422695.3 422846.0 422907.7 0.104 0.016 100.00 100.00
8GB simple numa-1-buffers none 420127.755712 421044.4 421598.3 422144.6 423093.7 423777.7 0.104 0.012 100.00 100.00
8GB simple numa-1-buffers random 408125.575066 410247.1 410399.4 410546.2 410731.1 410828.6 0.107 0.026 100.00 100.00
8GB simple numa-1-buffers colocated 437088.170225 438055.8 438324.7 438712.4 438931.0 439053.4 0.100 0.015 100.00 100.00
8GB simple numa-2-localalloc none 417427.169344 418270.3 418838.8 419372.3 420079.2 421378.3 0.105 0.012 100.00 100.00
8GB simple numa-2-localalloc random 407733.087076 409819.9 409989.2 410141.1 410349.4 410443.4 0.108 0.027 100.00 100.00
8GB simple numa-2-localalloc colocated 427076.145833 428141.6 428248.9 428421.9 428757.4 428942.7 0.103 0.016 100.00 100.00
8GB simple numa-3-no-tail none 416423.171947 417375.8 417944.6 418458.9 419220.5 419577.5 0.105 0.011 100.00 100.00
8GB simple numa-3-no-tail random 405829.718937 407883.1 408020.0 408131.0 408272.2 408496.8 0.108 0.026 100.00 100.00
8GB simple numa-3-no-tail colocated 421424.397567 422528.8 422703.8 422879.9 423100.0 423270.6 0.104 0.016 100.00 100.00
8GB simple numa-4-freelist none 419076.519833 420039.6 420611.5 421108.1 421851.3 422333.8 0.105 0.013 100.00 100.00
8GB simple numa-4-freelist random 399923.942600 402011.0 402182.1 402292.0 402462.0 402593.4 0.110 0.039 100.00 100.00
8GB simple numa-4-freelist colocated 408285.506814 409311.0 409527.0 409703.4 409848.1 409933.5 0.107 0.020 100.00 100.00
8GB simple numa-5-clocksweep none 418271.982316 419265.3 419777.1 420283.6 421177.6 421672.7 0.105 0.013 50.01 100.00
8GB simple numa-5-clocksweep random 412614.475308 414931.1 415027.7 415133.5 415266.1 415341.0 0.106 0.028 25.03 100.00
8GB simple numa-5-clocksweep colocated 413530.769663 414614.5 414791.1 414952.4 415109.0 415166.7 0.106 0.016 50.01 100.00
8GB simple numa-6-pgproc none 419625.446067 420662.8 421143.3 421698.9 422431.1 423015.8 0.105 0.013 25.02 100.00
8GB simple numa-6-pgproc random 413488.088807 415658.5 415842.1 415954.9 416118.8 416210.9 0.106 0.027 25.02 100.00
8GB simple numa-6-pgproc colocated 432241.958142 433413.7 433575.1 433732.2 434047.5 434135.9 0.101 0.015 50.01 100.00
8GB simple numa-7-pinning none 418205.981542 419096.8 419709.3 420246.4 421098.7 421900.0 0.105 0.013 25.02 100.00
8GB simple numa-7-pinning random 412783.906834 414853.9 415074.2 415432.1 415604.5 415712.6 0.106 0.027 50.01 100.00
8GB simple numa-7-pinning colocated 430539.555394 431708.8 431876.3 432020.8 432187.0 432274.7 0.102 0.017 25.02 100.00
8GB prepared numa-0-master none 703459.264267 704910.3 705975.2 707312.5 709332.2 710066.3 0.062 0.009 100.00 100.00
8GB prepared numa-0-master random 674244.112349 678564.5 678987.8 679454.4 680116.8 680501.2 0.065 0.025 100.00 100.00
8GB prepared numa-0-master colocated 714527.617635 716302.2 717010.7 717832.4 718839.7 719259.2 0.061 0.013 100.00 100.00
8GB prepared numa-1-buffers none 698173.827414 699533.4 700822.1 701923.8 703435.2 705671.6 0.063 0.009 100.00 100.00
8GB prepared numa-1-buffers random 673434.966316 677560.5 678049.7 678474.9 679062.0 679334.1 0.065 0.025 100.00 100.00
8GB prepared numa-1-buffers colocated 751566.117310 753320.4 754203.6 754831.3 755884.6 756464.8 0.058 0.012 100.00 100.00
8GB prepared numa-2-localalloc none 696928.357767 697891.6 699404.0 700916.8 702801.7 703614.1 0.063 0.009 100.00 100.00
8GB prepared numa-2-localalloc random 679125.597405 683401.6 683828.2 684343.2 684999.9 685223.4 0.064 0.024 100.00 100.00
8GB prepared numa-2-localalloc colocated 725088.057229 727100.1 727495.3 727966.1 728321.7 728666.5 0.060 0.011 100.00 100.00
8GB prepared numa-3-no-tail none 705127.816307 706515.8 707780.1 708851.5 710495.0 711812.6 0.062 0.009 100.00 100.00
8GB prepared numa-3-no-tail random 646426.356911 650349.4 650679.0 651084.4 651568.6 651890.3 0.068 0.025 100.00 100.00
8GB prepared numa-3-no-tail colocated 778274.192406 780341.2 780852.1 781509.3 782091.3 782312.0 0.056 0.012 100.00 100.00
8GB prepared numa-4-freelist none 688843.746004 690237.4 691476.0 692486.0 694485.6 695673.8 0.064 0.010 100.00 100.00
8GB prepared numa-4-freelist random 677409.346556 681680.9 682064.1 682463.7 683152.7 683382.7 0.065 0.029 100.00 100.00
8GB prepared numa-4-freelist colocated 777418.735444 779666.9 780072.2 780461.8 781045.9 781351.5 0.056 0.015 100.00 100.00
8GB prepared numa-5-clocksweep none 710909.011736 712640.0 713743.3 715092.0 716672.9 717912.9 0.062 0.010 50.01 100.00
8GB prepared numa-5-clocksweep random 693200.545485 697827.0 698333.0 698726.5 699376.2 699820.8 0.063 0.025 50.01 100.00
8GB prepared numa-5-clocksweep colocated 806920.631855 808679.8 809650.2 810959.6 811889.6 812648.1 0.054 0.013 25.02 100.00
8GB prepared numa-6-pgproc none 700719.229454 701968.1 703513.1 704830.2 706751.1 708149.9 0.063 0.010 25.02 100.00
8GB prepared numa-6-pgproc random 678244.443613 682195.4 682688.0 683070.6 683702.7 683965.3 0.065 0.025 25.02 100.00
8GB prepared numa-6-pgproc colocated 776707.680192 778630.0 779342.8 780372.5 781325.5 781939.7 0.056 0.013 25.02 100.00
8GB prepared numa-7-pinning none 707324.231529 708769.6 709673.6 710912.0 712442.2 713431.9 0.062 0.011 25.02 100.00
8GB prepared numa-7-pinning random 680872.484194 684928.7 685481.9 686003.1 686701.9 686972.7 0.064 0.025 25.02 100.00
8GB prepared numa-7-pinning colocated 784736.528912 786979.4 787586.1 788031.7 788616.5 788929.4 0.056 0.012 25.02 100.00
32GB simple numa-0-master none 481181.594955 485356.2 486575.9 487827.3 489003.9 489647.1 0.091 0.012 61.83 99.65
32GB simple numa-0-master random 466792.988100 472495.3 472962.0 473188.3 473459.6 473618.0 0.094 0.021 64.78 99.65
32GB simple numa-0-master colocated 482942.015715 486958.0 487531.7 488433.8 488592.6 488723.1 0.091 0.016 65.09 99.65
32GB simple numa-1-buffers none 480876.817874 485096.0 486160.2 486995.0 488304.7 489315.1 0.091 0.011 64.91 99.65
32GB simple numa-1-buffers random 460639.013021 466104.2 466494.0 466585.7 466694.7 466763.1 0.095 0.022 65.08 99.65
32GB simple numa-1-buffers colocated 488900.936303 493363.3 494035.3 494210.4 494426.5 494581.4 0.090 0.015 64.56 99.65
32GB simple numa-2-localalloc none 480484.228215 483794.7 485110.6 486269.1 487618.9 488064.9 0.091 0.011 64.68 99.65
32GB simple numa-2-localalloc random 459902.532232 464932.4 465444.2 465666.4 465843.8 465979.5 0.095 0.021 63.98 99.65
32GB simple numa-2-localalloc colocated 480102.371099 484327.4 484649.3 484841.3 485125.5 485420.4 0.091 0.014 63.97 99.65
32GB simple numa-3-no-tail none 471143.298591 474607.7 475852.4 476722.5 477940.3 478910.2 0.093 0.025 64.27 99.65
32GB simple numa-3-no-tail random 455567.453537 461096.7 461524.9 461848.0 462028.8 462150.5 0.096 0.022 64.21 99.65
32GB simple numa-3-no-tail colocated 491308.961289 495301.4 495628.7 495804.8 495992.0 496112.7 0.089 0.014 65.19 99.65
32GB simple numa-4-freelist none 480167.819014 483653.4 484628.6 485671.5 486689.1 487170.1 0.091 0.059 64.82 99.65
32GB simple numa-4-freelist random 472777.817626 478481.4 478987.1 479268.7 479542.4 479691.1 0.093 0.073 63.91 99.65
32GB simple numa-4-freelist colocated 492657.381699 496841.9 497470.7 497645.4 497820.0 497909.6 0.089 0.015 64.26 99.65
32GB simple numa-5-clocksweep none 483805.882520 489653.3 491572.4 492783.5 494432.2 495466.7 0.091 0.018 50.00 99.64
32GB simple numa-5-clocksweep random 468040.122668 474260.0 478357.7 478606.9 478963.5 479277.5 0.094 0.028 50.00 99.64
32GB simple numa-5-clocksweep colocated 482304.331109 486744.5 492023.4 492340.4 492606.3 492837.7 0.091 0.022 50.00 99.63
32GB simple numa-6-pgproc none 484479.447307 493097.9 494476.8 495464.5 496709.8 497211.6 0.090 0.027 25.00 99.64
32GB simple numa-6-pgproc random 457878.575808 464597.3 468862.4 469154.5 469344.3 469447.5 0.096 0.033 25.01 99.64
32GB simple numa-6-pgproc colocated 457805.057399 465111.0 467395.5 467654.0 467881.0 467959.6 0.096 0.028 25.01 99.64
32GB simple numa-7-pinning none 475346.501704 482907.0 485029.7 486132.5 487681.2 488656.5 0.092 0.024 25.00 99.64
32GB simple numa-7-pinning random 465628.120519 468806.0 476067.9 477194.0 477610.8 477786.0 0.094 0.029 50.00 99.64
32GB simple numa-7-pinning colocated 492061.233271 498352.0 501791.5 502016.0 502447.6 502844.3 0.089 0.023 50.00 99.64
32GB prepared numa-0-master none 883118.725149 884270.1 888725.8 891838.8 895229.1 897796.6 0.050 0.006 83.01 99.65
32GB prepared numa-0-master random 883127.213183 890679.6 891330.5 891699.5 892747.5 892965.9 0.050 0.014 83.94 99.65
32GB prepared numa-0-master colocated 1015416.796294 1019694.9 1020803.6 1023038.3 1023852.4 1024406.0 0.043 0.008 83.64 99.65
32GB prepared numa-1-buffers none 914220.121625 915460.5 918915.1 922621.8 926953.8 930621.5 0.048 0.005 83.77 99.65
32GB prepared numa-1-buffers random 871912.362629 879456.8 880058.4 881118.8 881811.5 882059.6 0.050 0.015 81.67 99.65
32GB prepared numa-1-buffers colocated 1049240.181634 1054167.4 1055629.3 1057257.7 1059329.6 1061070.9 0.042 0.008 82.07 99.65
32GB prepared numa-2-localalloc none 887616.565291 889474.7 892733.3 895617.5 900252.9 902337.3 0.049 0.005 82.84 99.65
32GB prepared numa-2-localalloc random 870508.549070 878103.2 878768.6 879038.5 879350.0 879493.6 0.050 0.015 82.16 99.65
32GB prepared numa-2-localalloc colocated 1020867.691158 1025611.1 1026682.9 1027615.3 1028681.4 1029457.2 0.043 0.008 82.23 99.65
32GB prepared numa-3-no-tail none 885775.797641 887798.4 891346.1 893870.0 897065.2 900503.7 0.049 0.005 83.08 99.65
32GB prepared numa-3-no-tail random 869874.940118 878303.4 879123.8 879556.0 880170.8 880301.7 0.050 0.015 81.38 99.65
32GB prepared numa-3-no-tail colocated 1023551.259812 1027846.4 1029726.4 1031542.5 1033057.1 1034226.2 0.043 0.008 82.18 99.65
32GB prepared numa-4-freelist none 888068.163937 890917.9 893654.9 896889.9 900299.0 903962.8 0.049 0.089 82.15 99.65
32GB prepared numa-4-freelist random 871501.689081 879536.5 880054.8 880596.9 881309.3 881502.9 0.050 0.054 81.62 99.65
32GB prepared numa-4-freelist colocated 1024148.485277 1030793.9 1031629.6 1032165.3 1032711.2 1033216.2 0.043 0.015 82.00 99.65
32GB prepared numa-5-clocksweep none 918069.797166 933014.9 936907.0 940382.8 944285.4 949159.3 0.048 0.018 25.00 99.64
32GB prepared numa-5-clocksweep random 885036.283920 894589.0 903520.2 905257.8 906320.4 906490.0 0.049 0.024 50.00 99.64
32GB prepared numa-5-clocksweep colocated 994162.961936 1011384.1 1018072.8 1020455.0 1021685.1 1022622.2 0.044 0.028 25.00 99.64
32GB prepared numa-6-pgproc none 907854.601956 919981.4 924082.5 928039.6 932036.9 934123.2 0.048 0.014 25.00 99.64
32GB prepared numa-6-pgproc random 879230.681109 894617.4 897742.0 898927.7 899484.6 899735.2 0.050 0.025 50.00 99.64
32GB prepared numa-6-pgproc colocated 1002350.405064 1016951.3 1022562.5 1023483.4 1025207.8 1025905.2 0.044 0.020 50.00 99.64
32GB prepared numa-7-pinning none 871253.230579 882872.7 885748.8 887650.3 890516.4 892967.4 0.050 0.014 50.00 99.64
32GB prepared numa-7-pinning random 846590.347022 854119.9 868664.9 874487.6 875205.1 875575.1 0.052 0.027 25.00 99.64
32GB prepared numa-7-pinning colocated 974530.487611 993520.8 995697.0 996657.9 998134.0 998447.0 0.045 0.022 26.91 99.64
[application/pdf] numa-xeon-e5-2699.pdf (53.6K, ../[email protected]/12-numa-xeon-e5-2699.pdf)
download
[application/pdf] numa-hb176.pdf (54.2K, ../[email protected]/13-numa-hb176.pdf)
download
view thread (89+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox