Feedback-ID: i675e48f3:Fastmail
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.600.51.1.1\))
Subject: Re: Adding basic NUMA awareness
From: "Burd, Greg" <greg@burd.me>
In-Reply-To: <ndvygkpdx44pmi4xbkf52gfrl77cohpefr42tipvd5dgiaeuyd@fe2og2kxyjnc>
Date: Thu, 10 Jul 2025 08:13:43 -0400
Cc: Tomas Vondra <tomas@vondra.me>,
 Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Dmitry Dolgov <9erthalion6@gmail.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <628EE169-6901-466E-9191-B33DBAB05B26@burd.me>
References: <uaqt63a2kz2g3qk2ujr55gdvmymnak3zsplcgrlh63ekbrytbc@a2wy22p3slge>
 <DC1A4B14-70B7-4B44-AD0B-E05D3292D833@getmailspring.com>
 <ndvygkpdx44pmi4xbkf52gfrl77cohpefr42tipvd5dgiaeuyd@fe2og2kxyjnc>
To: Andres Freund <andres@anarazel.de>
Archived-At: 
 <https://www.postgresql.org/message-id/628EE169-6901-466E-9191-B33DBAB05B26%40burd.me>
Precedence: bulk


> On Jul 9, 2025, at 1:23=E2=80=AFPM, Andres Freund <andres@anarazel.de> =
wrote:
>=20
> Hi,
>=20
> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>> On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:
>>=20
>>> FWIW, I've started to wonder if we shouldn't just get rid of the =
freelist
>>> entirely. While clocksweep is perhaps minutely slower in a single
>>> thread than
>>> the freelist, clock sweep scales *considerably* better [1]. As it's =
rather
>>> rare to be bottlenecked on clock sweep speed for a single thread
>>> (rather then
>>> IO or memory copy overhead), I think it's worth favoring clock =
sweep.
>>=20
>> Hey Andres, thanks for spending time on this.  I've worked before on
>> freelist implementations (last one in LMDB) and I think you're onto
>> something.  I think it's an innovative idea and that the speed
>> difference will either be lost in the noise or potentially entirely
>> mitigated by avoiding duplicate work.
>=20
> Agreed. FWIW, just using clock sweep actually makes things like DROP =
TABLE
> perform better because it doesn't need to maintain the freelist =
anymore...
>=20
>=20
>>> Also needing to switch between getting buffers from the freelist and
>>> the sweep
>>> makes the code more expensive.  I think just having the buffer in =
the sweep,
>>> with a refcount / usagecount of zero would suffice.
>>=20
>> If you're not already coding this, I'll jump in. :)
>=20
> My experimental patch is literally a four character addition ;), =
namely adding
> "0 &&" to the relevant code in StrategyGetBuffer().
>=20
> Obviously a real patch would need to do some more work than that.  =
Feel free
> to take on that project, I am not planning on tackling that in near =
term.
>=20

I started on this last night, making good progress.  Thanks for the =
inspiration.  I'll create a new thread to track the work and =
cross-reference when I have something reasonable to show (hopefully =
later today).

> There's other things around this that could use some attention. It's =
not hard
> to see clock sweep be a bottleneck in concurrent workloads - partially =
due to
> the shared maintenance of the clock hand. A NUMAed clock sweep would =
address
> that.

Working on it.  Other than NUMA-fying clocksweep there is a function =
have_free_buffer() that might be a tad tricky to re-implement =
efficiently and/or make NUMA aware.  Or maybe I can remove that too?  It =
is used in autoprewarm.c and possibly other extensions, but no where =
else in core.

> However, we also maintain StrategyControl->numBufferAllocs, which is a
> significant contention point and would not necessarily be removed by a
> NUMAificiation of the clock sweep.

Yep, I noted this counter and its potential for contention too.  =
Fortunately, it seems like it is only used so that "bgwriter can =
estimate the rate of buffer consumption" which to me opens the door to a =
less accurate partitioned counter, perhaps something lock-free (no =
mutex/CAS) that is bucketed then combined when read.

A quick look at bufmgr.c indicates that recent_allocs (which is =
StrategyControl->numBufferAllocs) is used to track a "moving average" =
and other voodoo there I've yet to fully grok.  Any thoughts on this =
approximate count approach?

Also, what are your thoughts on updating the algorithm to CLOCK-Pro [1] =
while I'm there?  I guess I'd have to try it out, measure it a lot and =
see if there are any material benefits.  Maybe I'll keep that for a =
future patch, or at least layer it... back to work!

> Greetings,
>=20
> Andres Freund

best.

-greg

[1] =
https://www.usenix.org/legacy/publications/library/proceedings/usenix05/te=
ch/general/full_papers/jiang/jiang_html/html.html