Message-ID: <a7ce37b4-da5e-4565-a9c3-4f92d7098128@data-bene.io>
Date: Wed, 9 Jul 2025 06:40:00 +0000
MIME-Version: 1.0
Subject: Re: Adding basic NUMA awareness - Preliminary feedback and outline
 for an extensible approach
To: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <099b9433-2855-4f1b-b421-d078a5d82017@vondra.me>
 <1caea3b1-3e0b-4bdb-a630-75671f1f837a@data-bene.io>
 <cwp57ttiqplhbobl74obmry4gjwtqhdh5omm7fhwgdvft4qg4q@jmpveobozjhj>
 <d630b9fd-d7eb-4642-8c1c-16be295bdace@data-bene.io>
 <949e555b-84ed-4f8f-863e-f88a15781142@vondra.me>
 <ececf471-0d15-46c7-8d97-82ef49a628cb@data-bene.io>
 <89c1f26c-977f-44e2-9d78-ddff7c8268b2@vondra.me>
From: =?UTF-8?Q?C=C3=A9dric_Villemain?= <cedric.villemain@data-bene.io>
Autocrypt: addr=cedric.villemain@data-bene.io; keydata=
 xsBNBEvW1WwBCAC6CG3zZnRpQwmSiFoTjaHlgt7RQ2RmYDqSB7DNtOItV9bBDUe4DKxV9iNF
 SoD2yV2LrzsYghWXwI0snHGVGCLkJFYBkqyMnJeli5PoHcn9jI12mOz0hdpsTmTJbH+JCLXF
 admIjh+sz+m7X4net3UQhj8tCaExkdNsw7FWfNr6oHOVkLWlTWh1df1dfTZ3CBZ8vc1L6dIn
 mWe3i2sRGQ/fCy7XfLqjFe5U+D3jXmqF92p7n8v0vwDZarn4MVeBegTtBYUxpM8Bj78Lga9a
 SW+ojYo21Zk/TYn+fdOX4L9nNiRFgnAYVjPlWwjV30WpVlQ56o8U+zV2hCNRQyl5QxkNABEB
 AAHNMUPDqWRyaWMgVmlsbGVtYWluIDxjZWRyaWMudmlsbGVtYWluQGRhdGEtYmVuZS5pbz7C
 wI4EEwEKADgWIQQ66LJpfJwdoY4rdHTbIB9Ms1QXmwUCY3ARzwIbAwULCQgHAgYVCgkICwIE
 FgIDAQIeAQIXgAAKCRDbIB9Ms1QXm9oTB/41lFaXCcwtzLjjN7Qr7lImBjiTOntvhzFOBUMP
 xZWmrXzVl1jVzQm8ubsZNJFB98kw5MrhZjY0sY4x6Txlh6y/OTtpT0RWfZQOGSmfpMq8QDuV
 gymCfXyr0t4XHFou31nlaqEz1CyUi8ZaYdutJAV0L8j1e0LUHvTcCmlHOsb1uS+Xf5yc7ub+
 eRiGtjXwM+OJBYYkFZ2Ebyw7mHWRF4+B3QXUjZhPCQZWBQE1GeX1KxrSCfSKfqujVWCwgiF9
 q5TQisgzVRwPBwsi5Fam2X7UHD8I6ZjaAiKHJXezP/Ied/bup457DlXfOg5ORePJgIldMo0p
 AtHIY1061TckCPs9zsBNBEvW1WwBCAC1iEAHk3lEWEDp9O/DebejwyNr9Zmun6NXPN5axuSj
 5VKWP/2CCmsPkcIQlH70rEtZl4vQjXrdRviRcc56TC/SSXti8uOVr/55uStitTCjKRkpxPdf
 ReOFuOB7OUkHrE3gF6N0UNgOMgmBsT5+rgFm+xqY6TOrvnDVGUbAGkJIdzgTjzeu96BcBEuF
 59r+grsmff8PUGtertKeXBr8NlliYZEFCilqus3PheOetWewysHROQJrqnSkpRP7zJWyFYY5
 W6PWKQlUgcItrWsgPlHl5gmoH68zesTlW1rX1Z56V8vLPF7RTAPG6g1J7l29wLxUYTwF5Y64
 gy+jXAEUJHKZABEBAAHCwF8EGAECAAkFAkvW1WwCGwwACgkQ2yAfTLNUF5ubKAgAtZdh0QJU
 Wc7WfZhoGOcAEADZJ9n6GSuhL+oxuhPfym1zsU70241rKYZH1m53CWWo+aNyThCXJx+HSSRV
 /hLNVW34DErJeNJIYwdxPo5/AjOhIUr4e8gMYuviFESBaMpgl8mSM1B1cHVBMZcsQb43Rc3V
 VfIMNRc4NB16uc7Sm++fbraQq13rB8HC6uYu7PbEAdX3C0IokR3JGb1NghQKga/OoU22KPEx
 Y+bFk4h3vq9MjlKLir0UJZHKSQ5xec6RcehoMBy0FHBEuK/72xN36dR/3bVwyeR76S7xf9UQ
 gXdpciBG95/tLRcpelwx8GaRcivTGc0FCBUzqG/YcCqvcA==
In-Reply-To: <89c1f26c-977f-44e2-9d78-ddff7c8268b2@vondra.me>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/a7ce37b4-da5e-4565-a9c3-4f92d7098128%40data-bene.io>
Precedence: bulk

> On 7/8/25 18:06, C=C3=A9dric Villemain wrote:
>>
>>
>>
>>
>>
>>
>>> On 7/8/25 03:55, C=C3=A9dric Villemain wrote:
>>>> Hi Andres,
>>>>
>>>>> Hi,
>>>>>
>>>>> On 2025-07-05 07:09:00 +0000, C=C3=A9dric Villemain wrote:
>>>>>> In my work on more careful PostgreSQL resource management, I've co=
me
>>>>>> to the
>>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>>> integrating
>>>>>> NUMA-specific management directly into core PostgreSQL in such a w=
ay.
>>>>>
>>>>> I think it's actually the opposite - whenever we pushed stuff like =
this
>>>>> outside of core it has hurt postgres substantially. Not having
>>>>> replication in
>>>>> core was a huge mistake. Not having HA management in core is
>>>>> probably the
>>>>> biggest current adoption hurdle for postgres.
>>>>>
>>>>> To deal better with NUMA we need to improve memory placement and
>>>>> various
>>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>>> to do
>>>>> outside of core.
>>>>
>>>> Except the backend pinning which is easy to achieve, thus my comment=
 on
>>>> the related patch.
>>>> I'm not claiming NUMA memory and all should be managed outside of co=
re
>>>> (though I didn't read other patches yet).
>>>>
>>>
>>> But an "optimal backend placement" seems to very much depend on where=
 we
>>> placed the various pieces of shared memory. Which the external module
>>> will have trouble following, I suspect.
>>>
>>> I still don't have any idea what exactly would the external module do=
,
>>> how would it decide where to place the backend. Can you describe some
>>> use case with an example?
>>>
>>> Assuming we want to actually pin tasks from within Postgres, what I
>>> think might work is allowing modules to "advise" on where to place th=
e
>>> task. But the decision would still be done by core.
>>
>> Possibly exactly what you're doing in proc.c when managing allocation =
of
>> process, but not hardcoded in postgresql (patches 02, 05 and 06 are go=
od
>> candidates), I didn't get that they require information not available =
to
>> any process executing code from a module.
>>
>=20
> Well, it needs to understand how some other stuff (especially PGPROC
> entries) is distributed between nodes. I'm not sure how much of this
> internal information we want to expose outside core ...
>=20
>> Parts of your code where you assign/define policy could be in one or
>> more relevant routines of a "numa profile manager", like in an
>> initProcessRoutine(), and registered in pmroutine struct:
>>
>> pmroutine =3D GetPmRoutineForInitProcess();
>> if (pmroutine !=3D NULL &&
>>  =C2=A0=C2=A0=C2=A0 pmroutine->init_process !=3D NULL)
>>  =C2=A0=C2=A0=C2=A0 pmroutine->init_process(MyProc);
>>
>> This way it's easier to manage alternative policies, and also to be ab=
le
>> to adjust when hardware and linux kernel changes.
>>
>=20
> I'm not against making this extensible, in some way. But I still
> struggle to imagine a reasonable alternative policy, where the external
> module gets the same information and ends up with a different decision.
>=20
> So what would the alternate policy look like? What use case would the
> module be supporting?


That's the whole point: there are very distinct usages of PostgreSQL in=20
the field. And maybe not all of them will require the policy defined by=20
PostgreSQL core.

May I ask the reverse: what prevent external modules from taking those=20
decisions ? There are already a lot of area where external code can take=20
over PostgreSQL processing, like Neon is doing.

There are some very early processing for memory setup that I can see as=20
a current blocker, and here I'd refer a more compliant NUMA api as=20
proposed by Jakub so it's possible to arrange based on workload,=20
hardware configuration or other matters. Reworking to get distinct=20
segment and all as you do is great, and combo of both approach probably=20
of great interest. There is also this weighted interleave discussed and=20
probably much more to come in this area in Linux.

I think some points raised already about possible distinct policies, I=20
am precisely claiming that it is hard to come with one good policy with=20
limited setup options, thus requirement to keep that flexible enough=20
(hooks, api, 100 GUc ?).

There is an EPYC story here also, given the NUMA setup can vary=20
depending on BIOS setup, associated NUMA policy must probably take that=20
into account (L3 can be either real cache or 4 extra "local" NUMA nodes=20
- with highly distinct access cost from a RAM module).
Does that change how PostgreSQL will place memory and process? Is it=20
important or of interest ?


--=20
C=C3=A9dric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D