Feedback-ID: i675e48f3:Fastmail
MIME-Version: 1.0
Date: Tue, 15 Sep 2026 16:41:44 -0400
From: "Greg Burd" <greg@burd.me>
To: "Jakub Wartak" <jakub.wartak@enterprisedb.com>
Cc: "Andres Freund" <andres@anarazel.de>,
 "Alexey Makhmutov" <a.makhmutov@postgrespro.ru>,
 "PostgreSQL Hackers" <pgsql-hackers@lists.postgresql.org>,
 "Tomas Vondra" <tomas@vondra.me>
Message-Id: <b3c09a66-b2f6-49a6-a2fe-bbfe0cda3e6f@app.fastmail.com>
In-Reply-To: <f78dfe7d-4a37-40ec-8e60-c2cc47c48364@app.fastmail.com>
References: <5bc6d309-61fe-4d8b-9e1f-a2961564a559@vondra.me>
 <05df16f8-025a-43cd-9636-3194464012ed@vondra.me>
 <CAKZiRmxwN+qMpbijCLPix_y6mwSjgus2CPPj=1+uFo9fQG-Knw@mail.gmail.com>
 <403c813e-7d63-4ee8-a95f-4e5c1e310f4d@vondra.me>
 <uezi46xhhbvdjgdi6wl7iqgfcdh4jmnnyzbfovdcrck6ywqa7j@fj3yimxvekk6>
 <0e1b997d-99c8-40f4-bc32-6c044bc7ed9a@vondra.me>
 <zndz3dlmp7xlypczxjbwdfrey3masto6vuwpnzjvgunslprv25@rucfkdgfpb4p>
 <rsjxzhnxxfq5i5yxv66mhinb42o3vzmqpkbfpexpkk5prreh2l@jyp73gsyyfzn>
 <2db78610-b480-4aa0-a1b6-57f1c2dcb708@vondra.me>
 <clx4zzd7kau4vvh5ynu5ssxg3jqfqzurgcbtotytzgzkhb3nis@qfl5xwv44yad>
 <2e1be441-5608-4bc8-9ac4-6fad9a060db4@vondra.me>
 <334f7644-6716-4e63-a7c4-13c5cbcdd210@vondra.me>
 <CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay+mOEKs9rzCkegUA@mail.gmail.com>
 <86e7e218-9013-4cdb-9ed2-dfda49640b4d@vondra.me>
 <CAKZiRmzumFJ8+Bqz5mfX+QGkrKEaH32zz7kPgEShHWXE4QzyQw@mail.gmail.com>
 <c3954e28-0c38-4df8-b76e-ec09f6c04021@vondra.me>
 <CAKZiRmwsjwDatuV5JfUaAbkTxjnUaL_UK3SkFTJee-jo-u_N7Q@mail.gmail.com>
 <2b6abd8f-3659-4be1-9499-b4f00f973f22@vondra.me>
 <CAKZiRmzigM9XOz6m0K070ZVSmaKpjnVR4CZZENkfeH-rNv78wA@mail.gmail.com>
 <CAKZiRmz+0tYkoMAKq4Qoc6M1-ZhYFEJnJO23Q2Kf8eyhx3S4og@mail.gmail.com>
 <CAKZiRmzYM_5e8wStGQVVr7_v62br-EbUbTecK6HBFAKOGMBCfQ@mail.gmail.com>
 <CAKZiRmzT1M1fEFf5fRq2Qg3Vz0wLbdY+uNdF-0ORhDTUOUVhcA@mail.gmail.com>
 <f78dfe7d-4a37-40ec-8e60-c2cc47c48364@app.fastmail.com>
Subject: Re: Adding basic NUMA awareness
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/b3c09a66-b2f6-49a6-a2fe-bbfe0cda3e6f%40app.fastmail.com>
Precedence: bulk


On Thu, Sep 10, 2026, at 10:50 AM, Greg Burd wrote:
> The obvious experiment
> is a 3-way on one box: master / your optimized / your optimized + our
> batched claim. I have your harness working now and I'm happy to run that
> on both a 2-node and a 6-node box and post the numbers. If you'd rather
> I test a particular partition count or the aligned variant alongside it,
> say so and I'll fold it in.  Too late, I've started the test... ;-)

I said I'd run the three-way comparison: master vs your patchset vs
your patchset plus our batched claim, on your methodology, on both a
2-node and a 6-node box.

Setup, same as before what you've done most recently but I did 5 runs
per point instead of 3 and rotated the arm order each run, because the
variance turned out to matter (more on that below). Fresh cluster per
arm per run. All numbers are medians of 5.

Arms:
  master     3086e547
  V+W        v20260605 0001-0006 + your 0008
  V+W+batch  the above, plus the batched per-partition hand claim

Boxes:
  2 nodes -- i4i.metal, 128 threads, datadir on local NVMe RAID0
  6 nodes -- r8i.metal-96xl, 384 threads, datadir on io2 EBS (40k IOPS)

Both patched arms ran with shared_buffers_numa=on and
debug_shmem_populate=on. Confirmed before measuring that partitioning
was actually live: 4 partitions on the 2-node box, 6 on the 6-node box,
covering all 1048576 buffers.

seqconcurrscans, median TPS of 5, spread is (max-min)/median

  2 nodes
   cl    master      V+W       V+W+batch   V+W vs mst  batch vs V+W
    1   1.7 +-6%   1.5 +-11%   1.6 +-9%      -12.3%       +10.2%
    8   5.7 +-2%   6.3 +-20%   6.5 +-9%      +11.9%        +3.5%
   32  18.8 +-5%  19.9 +-6%   19.9 +-7%       +5.5%        +0.4%
   64  34.8 +-3%  37.6 +-7%   38.4 +-5%       +8.0%        +2.1%

  6 nodes
   cl    master      V+W       V+W+batch   V+W vs mst  batch vs V+W
    1   1.7 +-7%   0.9 +-20%   0.8 +-15%     -48.7%        -6.4%
    8   3.8 +-7%   4.7 +-19%   4.9 +-18%     +23.3%        +4.4%
   32   9.0 +-6%  13.2 +-15%  13.3 +-8%      +46.4%        +1.1%
   64  14.6 +-7%  21.4 +-18%  22.3 +-9%      +46.9%        +4.2%

pgbenchS: flat within +-1% for all three arms at 2 nodes. At 6 nodes
there is something: V+W +0.5/+1.0/+4.0% at c=8/32/64, and V+W+batch
+3.3/+3.1/+7.0%. At c=64 that's master 975k, V+W 1.015M, V+W+batch
1.043M.

So, first this patchset is worth considerably more than the thread
currently credits it with. You measured ~9-10% on seqconcurrscans on
2-node i4i.metal and were disappointed; I get +5-12% on 2 nodes, which
matches you, but +23-47% on 6 nodes. The win scales with node count. So
the 9-10% isn't the ceiling, it's the floor of the range, and the 2-node
box is simply the wrong place to judge it. That may also be part of the
answer to your "why not 2-3x" question. Tomas's original numbers and
my earlier pgbenchS numbers were both on wider topologies than the box
you were testing on.

Second, on the composition question I raised: the two do compose, but
batching is clearly the junior partner once the hand is per-node.
+0.4-4.4% on seqconcurrscans on top of your patchset, +2-3% on pgbenchS
at 6 nodes. That's the honest result and it's smaller than I implied
when I suggested the experiment. The reason is visible in perf c2c,
records in the clock-sweep hot line at c=64:

  2 nodes:  master 6,389  ->  V+W 705  ->  V+W+batch 392
  6 nodes:  master 31,700 ->  V+W 739  ->  V+W+batch 786

Partitioning alone removes 89-98% of the traffic on that line. There
isn't much left for batching to take, which is exactly what the small
increment says. (Ignore the percentage columns in perf c2c output when
comparing across arms, they're shares within each run's own table, so
they're not comparable; the record counts are.)

Third, and I think this is the most important thing here: the c=1
regression is much worse at 6 nodes than at 2. -12% at 2 nodes, -49% at
6. Your residency explanation predicts this exactly. The per-partition
pool is s_b/nparts, so at 6 nodes each partition is 1.3GB against a
1.6GB relation and a single-client scan can never keep it resident.
Batching doesn't help and slightly hurts (-6.4%), which makes sense
since it doesn't change how the pool is divided. This looks like it
needs a floor: don't partition below some ratio of NBuffers to the
largest relation you expect to stay resident, rather than scaling
partitions with node count unconditionally. On the 6-node box the
default of one partition per node is already past that line.

Fourth, variance. Several patched-arm points have 15-32% spread between
min and max across 5 runs, worst at c=1 and generally worse at 6 nodes.
Single-run comparisons on seqconcurrscans aren't safe, which is probably
worth knowing before anyone draws conclusions from a one-shot run.

I'm happy to rerun any of it with different partition counts if that's
useful in particular I'd be glad to test a partition-size floor if you
want to try one, since the 6-node c=1 number is a concrete target to
aim at.

In the mean time I'll queue up the partitioning idea with my HOT/COLD
eviction algorithm but without the batching and see how that compares
just for fun.

best.

-greg