From: Jack Ng <Jack.Ng@huawei.com>
To: Andres Freund <andres@anarazel.de>, Dmitry Dolgov <9erthalion6@gmail.com>
CC: Thom Brown <thom@linux.com>, Ashutosh Bapat
	<ashutosh.bapat.oss@gmail.com>, Tomas Vondra <tomas@vondra.me>, Thomas Munro
	<thomas.munro@gmail.com>, PostgreSQL-development
	<pgsql-hackers@postgresql.org>, Ni Ku <jakkuniku@gmail.com>
Subject: RE: Changing shared_buffers without restart
Thread-Topic: Changing shared_buffers without restart
Thread-Index: 
 AQHb4czjArbykOnCnUaEQCqOWcY4q7QfGxSAgAJTTACAAPSigIADCKUAgAAFeACAC1ixAIAArMiAgAA2XgCAAARBgIAACBkAgAAIcACAAAIegIAAOSQAgAADOQCAAAGrAIAAAZKAgAAGWACAAAVUAIAABgaA///AHlA=
Date: Mon, 14 Jul 2025 15:10:50 +0000
Message-ID: <39b5afc524174c8a8fa6ef058676a596@huawei.com>
References: 
 <CAExHW5tztw-ApbcsAHv+=vTZTO6w9hC+Q7OtUEWWfEgvDFHWxA@mail.gmail.com>
 <nu3pggzvuqomroda5cliicehtuam4kd4sxfm27d5rqpejnqutf@micq2wwi66jc>
 <CAA-aLv7p=9jCy_-67+Wj2vrRL1QCV-X0ZFZmpAxBJqLPp-ho+A@mail.gmail.com>
 <s2eqkawv2m3knhtr32cut6zvyebas25cszpnmlh6amjw6fjjzf@jyzij4mrquqa>
 <ndv2aa7z6qxzp7xehoqi4xz2sujfm4mgwictalr5ovtfhwv2pp@fjzdg5pj3r6t>
 <pdhm6tcvwhmnodv7rnmev2ikd2jy47f72rnygt6majew2at62o@6ac7byvkpiko>
 <afnt6ptmwx5zef46wmvxnqa2e57fwh57yg53met3hrbmsswavr@i663vegqqcal>
 <vslqe4duatd5hp4cw6eom3il5umpfrixylmpoh2ar27x3rstel@iengs2ijyflq>
 <eww3iiu2b64oay6qnz3f2kptccc3fxljm4bas2a2i2l5jhi4cd@kojmlzsq5uq2>
 <pkjfmei3j6yqmdi76cwzsmn4z34zg5yanb7rfv2rq4aik6ija3@xq6nrkvpya76>
 <fkm2z3idtem4raycfvxhuxwytxkugcqhjiryzub2hstjkbjele@cn2v5fzvvj2k>
In-Reply-To: <fkm2z3idtem4raycfvxhuxwytxkugcqhjiryzub2hstjkbjele@cn2v5fzvvj2k>
Accept-Language: en-CA, zh-CN, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/39b5afc524174c8a8fa6ef058676a596%40huawei.com>
Precedence: bulk

If I understanding correctly, putting a new buffer in the freelist before u=
pdating NBuffers could break existing logic that calls BufferIsValid(bufnum=
) and asserts bufnum <=3D NBuffers? (since a backend can grab the new buffe=
r and checks its validity before the coordinator can add it to the freelist=
.)

But it seems updating NBuffers before adding new elements to the freelist c=
ould be problematic too?  Like if a new buffer is already chosen as a victi=
m and then the coordinator adds it to the freelist, would that lead to "dou=
ble-use"? (seems possible at least with current logic and serialization in =
StrategyGetBuffer). If that's a valid concern, would something like this wo=
rk?

1) initialize buffer headers, with a new state/flag to indicate "add-pendin=
g"
2) update NBuffers
  -- add a check in clock-sweep logic for "add-pending" and skip them
3) put them onto the freelist
4) when a new element is grabbed from freelist, check for and reset add-pen=
ding flag.=20

This ensure the new element is always obtained from the freelist first I th=
ink.

Jack

>-----Original Message-----
>From: Andres Freund <andres@anarazel.de>
>Sent: Monday, July 14, 2025 10:23 AM
>To: Dmitry Dolgov <9erthalion6@gmail.com>
>Cc: Thom Brown <thom@linux.com>; Ashutosh Bapat
><ashutosh.bapat.oss@gmail.com>; Tomas Vondra <tomas@vondra.me>;
>Thomas Munro <thomas.munro@gmail.com>; PostgreSQL-development <pgsql-
>hackers@postgresql.org>; Jack Ng <Jack.Ng@huawei.com>; Ni Ku
><jakkuniku@gmail.com>
>Subject: Re: Changing shared_buffers without restart
>
>Hi,
>
>On 2025-07-14 16:01:50 +0200, Dmitry Dolgov wrote:
>> > On Mon, Jul 14, 2025 at 09:42:46AM -0400, Andres Freund wrote:
>> > What on earth would be the point of putting a buffer on the freelist
>> > but not make it reachable by the clock sweep? To me that's just nonsen=
sical.
>>
>> To clarify, we're not talking about this scenario as "that's how it
>> would work after the resize". The point is that to expand shared
>> buffers they need to be initialized, included into the whole buffer
>> machinery (freelist, clock sweep, etc.) and NBuffers has to be updated.
>
>It seems pretty obvious to that the order has to be
>
>1) initialize buffer headers
>2) update NBuffers
>3) put them onto the freelist
>
>(with 3) hopefully becoming obsolete)
>
>
>> Those steps are separated in time, and I'm currently trying to
>> understand what are the consequences of performing them in different
>> order and whether there are possible concurrency issues under various
>> scenarios. Does this make more sense, or still not?
>
>I still don't understand why it'd ever make sense to put a buffer onto the=
 freelist
>before updating NBuffers first.
>
>Greetings,
>
>Andres Freund