Feedback-ID: ic6394509:Fastmail
MIME-Version: 1.0
Date: Sun, 26 Oct 2025 07:33:32 +0100
From: "Joel Jacobson" <joel@compiler.org>
To: "Chao Li" <li.evan.chao@gmail.com>,
 "Arseniy Mukhin" <arseniy.mukhin.dev@gmail.com>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
Message-Id: <55d24cbb-e9ef-491f-a99b-b3dbd7cecdf9@app.fastmail.com>
In-Reply-To: <0BCA1C2D-B92C-459E-B1A6-6D06BA4C62CF@gmail.com>
References: <6899c044-4a82-49be-8117-e6f669765f7e@app.fastmail.com>
 <165530.1752362320@sss.pgh.pa.us>
 <02a7cd37-e2fc-4212-8b19-f8c239c95fb8@app.fastmail.com>
 <e396ecf3-4227-4918-b9ff-e9568dcebcf0@app.fastmail.com>
 <b6a427fe-2ed6-4568-85d9-207a68172617@app.fastmail.com>
 <CAK80=jhmE40KVqQ3ho37MArS7cAED1p9m7uikDxcnDmqdW7t8A@mail.gmail.com>
 <96f00bf1-cc9d-4520-9d02-9e14e7767c88@app.fastmail.com>
 <30c2aa7d-dd6c-4b68-a2e4-f217a1a34acf@app.fastmail.com>
 <CA+hUKGLrMGkWDB0cwTa0RqD+AF7O-Ywgck8aVYKwOQnZgYRRug@mail.gmail.com>
 <0b4d402a-9ac2-4aa8-acf8-8231dbe579ea@app.fastmail.com>
 <c8e84bb2-d2e5-4a77-b547-3a93c5778e7e@app.fastmail.com>
 <3095599.1758644879@sss.pgh.pa.us>
 <0dc6a2cc-5216-4dc1-9dd2-430cafc6095b@app.fastmail.com>
 <E7D8A5CD-C445-4713-B6B7-6CC1D3A71161@gmail.com>
 <baf71275-9083-41be-ae13-f80821541a1a@app.fastmail.com>
 <52CC167F-763B-4ECA-B0B4-DAB381816828@gmail.com>
 <d35a114d-204a-4b40-9284-06749532b724@app.fastmail.com>
 <9186C6D0-F7A9-482A-9183-89E530B57E36@gmail.com>
 <da0996!!!!!!d8-ecf9-490d-bbd0-2a5bdb5b115a@app.fastmail.com>
 <D1CF7CA5-213A-4FC1-86DB-052D58904A0A@gmail.com>
 <cad78ff4-aae9-4c1c-8e16-ff0ccd4957f1@app.fastmail.com>
 <1073593.1759423179@sss.pgh.pa.us>
 <4bd5e6c4-6fa7-44bb-869d-59a32a331fa8@app.fastmail.com>
 <CAFY6G8dap-bCnAnMG-2Gzew8yv2Vbi9gsx9+yszKMmd57ygfvA@mail.gmail.com>
 <85828f29-e72e-4400-94f3-9a69bc8dc239@app.fastmail.com>
 <2495353.1759860890@sss.pgh.pa.us>
 <8aeae418-92a6-4bbd-9c06-9574c79e59f7@app.fastmail.com>
 <2531672.1759868124@sss.pgh.pa.us>
 <474efa78-337c-41cd-a73a-f845a0115109@app.fastmail.com>
 <2749343.1759949176@sss.pgh.pa.us>
 <8bfca2be-1ec0-4e15-aafb-0b7b661fe936@app.fastmail.com>
 <9eba307f-f2fb-48f0-9507-2e197f39ef9e@app.fastmail.com>
 <cde2a91d-d47a-43b5-ab27-f98aa739d38e@app.fastmail.com>
 <8c71183a-0d28-4bcf-a806-78446ff95404@app.fastmail.com>
 <1009807.1760476747@sss.pgh.pa.us>
 <1F7227F5-C33D-4E2C-8511-33F1468590D0@gmail.com>
 <0a5a20d3-4621-46b3-b2ab-903f63a20dea@app.fastmail.com>
 <6F913129-ABEF-4004-AAF3-F22FC34!29AE8@gmail.com>
 <cbabbb45-ec27-47f1-a8f3-eb29073e9aef@app.fastmail.com>
 <ceab2c54-3463-4029-a7d8-895a617bb457@app.fastmail.com>
 <1547585.1760645808@sss.pgh.pa.us>
 <b8df8040-1b15-4a1a-b5f6-74acebbd2d0c@app.fastmail.com>
 <CAE7r3MLivh1sHWF06hrVXkiQbw-KChPcQsh+9CheXprm5vRVMQ@mail.gmail.com>
 <14865EB6-0BF4-462B-9072-10BDAC10C052@gmail.com>
 <CAE7r3MK-3AOdh1mpZ8hw9h6F_i0D5RMoAy7CttnfCJRpB8GJDA@mail.gmail.com>
 <0BCA1C2D-B92C-459E-B1A6-6D06BA4C62CF@gmail.com>
Subject: Re: Optimize LISTEN/NOTIFY
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/55d24cbb-e9ef-491f-a99b-b3dbd7cecdf9%40app.fastmail.com>
Precedence: bulk

On Sun, Oct 26, 2025, at 05:11, Chao Li wrote:
> I figured out a way to resolve the race condition for alt3:
>
> * add an awakening flag for every listener, this flag is only set by=20
> listeners
> * add an advisory pos for every listener, similar to alt1
> * if a listener is awaken, notify only updates the listener=E2=80=99s =
advisory=20
> pos; otherwise directly advance its position.
> * If a running listener see current pos is behind advisory pos, then=20
> stop reading
>
> See more details in attach patch file, I added code comments for my=20
> changes. Now the TAP test won=E2=80=99t hit the race condition.=20
> ```
> # +++ tap check in src/test/authentication +++
> t/008_listen-pos-race.pl .. skipped: Injection points not supported by=20
> this build
> Files=3D1, Tests=3D0,  0 wallclock secs ( 0.00 usr  0.00 sys +  0.03 c=
usr =20
> 0.01 csys =3D  0.04 CPU)
> Result: NOTESTS
> ```
>
> And with my solution, listeners longer will still use local pos, so=20
> that no longer need to acquire notification lock in every loop.

This sounds promising, similar to what I had in mind. I was thinking
about the idea of using the advisoryPos only when the listening backend
is known to be running (which felt like it would need another shared
boolean field), and to move its pos field directly only when it's not
running, since if it's running we don't need to optimize for context
switching, since it's by definition already running.

What I wanted to investigate what all the concurrency situations
that we can imagine, i.e. to permutate all possible differences
we can think of into a truth table, and reason about each case.

The ones I can think of are, from the perspective of SignalBackends,
reasoning about a specific listening backend:

{is interested in the notifications, is not interested in the notificati=
ons} x
{wakeupPending=3Dfalse, wakeupPending=3Dtrue} x
{pos < queueHeadBeforeWrite, pos =3D=3D queueHeadBeforeWrite, pos > queu=
eHeadBeforeWrite, pos =3D=3D queueHeadAfterWrite, pos > queueHeadAfterWr=
ite} x
{is running, is not running}

This gives 2x2x5x2=3D40 states to reason about. Some of these combinatio=
ns
are probably impossible, I still think it would be good to include them
and explain why they are impossible.

> The patch stack is: v20 patch -> alt3 patch -> tap patch -> my patch.=20
> Please see if my solution works.
>
> I also made a tiny change in the TAP script to allow it to terminate g=
racefully.

I haven't looked at the code yet, tried to apply the patch but it fails:

shasum of files:
```
ca54ffa02ac54efd65acce0d09b18e630b5d7982  0001-optimize_listen_notify-v2=
0.patch
5755701bb0e7ac7a0cea3abab9d74a0001b7b63a  0002-optimize_listen_notify-v2=
0.patch
5819e23b5760023be70d2582207b72164904e952  0002-optimize_listen_notify-v2=
0-alt3.txt
33d700dc0b3288d46705e85d381cb564d99079d1  0001-TAP-test-with-listener-po=
s-race.patch.nocfbot
8ee716451bd5f85761b666712bdfd8b5d936f92d  fix-race.patch
```

Trying to apply them on top of current master (39dcfda2d23ac39f14ecf4b83=
e01eae85d07d9e5):

```
% git apply 0001-optimize_listen_notify-v20.patch
% git apply 0002-optimize_listen_notify-v20.patch
% git apply 0002-optimize_listen_notify-v20-alt3.txt
% git apply 0001-TAP-test-with-listener-pos-race.patch.nocfbot
% git apply fix-race.patch
fix-race.patch:100: indent with spaces.
					     (QUEUE_POS_PRECEDES(queueHeadBeforeWrite, pos) && QUEUE_POS_PR=
ECEDES(pos, queueHeadAfterWrite))) &&
error: patch failed: src/backend/commands/async.c:250
error: src/backend/commands/async.c: patch does not apply
error: patch failed: src/test/authentication/t/008_listen-pos-race.pl:8
error: src/test/authentication/t/008_listen-pos-race.pl: patch does not =
apply
```

I'll try to resolve it manually, but in case you're quicker to reply, I'=
m sending this now.

/Joel