Feedback-ID: ic6394509:Fastmail
MIME-Version: 1.0
Date: Thu, 15 Jan 2026 03:14:09 +0100
From: "Joel Jacobson" <joel@compiler.org>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>
Message-Id: <18521cfd-830e-4e8f-bcb0-eacee535480a@app.fastmail.com>
In-Reply-To: <422696.1768432145@sss.pgh.pa.us>
References: <6899c044-4a82-49be-8117-e6f669765f7e@app.fastmail.com>
 <165530.1752362320@sss.pgh.pa.us>
 <02a7cd37-e2fc-4212-8b19-f8c239c95fb8@app.fastmail.com>
 <e396ecf3-4227-4918-b9ff-e9568dcebcf0@app.fastmail.com>
 <b6a427fe-2ed6-4568-85d9-207a68172617@app.fastmail.com>
 <CAK80=jhmE40KVqQ3ho37MArS7cAED1p9m7uikDxcnDmqdW7t8A@mail.gmail.com>
 <96f00bf1-cc9d-4520-9d02-9e14e7767c88@app.fastmail.com>
 <30c2aa7d-dd6c-4b68-a2e4-f217a1a34acf@app.fastmail.com>
 <CA+hUKGLrMGkWDB0cwTa0RqD+AF7O-Ywgck8aVYKwOQnZgYRRug@mail.gmail.com>
 <0b4d402a-9ac2-4aa8-acf8-8231dbe579ea@app.fastmail.com>
 <c8e84bb2-d2e5-4a77-b547-3a93c5778e7e@app.fastmail.com>
 <3095599.1758644879@sss.pgh.pa.us>
 <0dc6a2cc-5216-4dc1-9dd2-430cafc6095b@app.fastmail.com>
 <E7D8A5CD-C445-4713-B6B7-6CC1D3A71161@gmail.com>
 <baf71275-9083-41be-ae13-f80821541a1a@app.fastmail.com>
 <52CC167F-763B-4ECA-B0B4-DAB381816828@gmail.com>
 <d35a114d-204a-4b40-9284-06749532b724@app.fastmail.com>
 <9186C6D0-F7A9-482A-9183-89E530B57E36@gmail.com>
 <da0996!!!!!!!!!!!d8-ecf9-490d-bbd0-2a5bdb5b115a@app.fastmail.com>
 <D1CF7CA5-213A-4FC1-86DB-052D58904A0A@gmail.com>
 <cad78ff4-aae9-4c1c-8e16-ff0ccd4957f1@app.fastmail.com>
 <1073593.1759423179@sss.pgh.pa.us>
 <4bd5e6c4-6fa7-44bb-869d-59a32a331fa8@app.fastmail.com>
 <CAFY6G8dap-bCnAnMG-2Gzew8yv2Vbi9gsx9+yszKMmd57ygfvA@mail.gmail.com>
 <85828f29-e72e-4400-94f3-9a69bc8dc239@app.fastmail.com>
 <2495353.1759860890@sss.pgh.pa.us>
 <8aeae418-92a6-4bbd-9c06-9574c79e59f7@app.fastmail.com>
 <2531672.1759868124@sss.pgh.pa.us>
 <474efa78-337c-41cd-a73a-f845a0115109@app.fastmail.com>
 <2749343.1759949176@sss.pgh.pa.us>
 <8bfca2be-1ec0-4e15-aafb-0b7b661fe936@app.fastmail.com>
 <9eba307f-f2fb-48f0-9507-2e197f39ef9e@app.fastmail.com>
 <cde2a91d-d47a-43b5-ab27-f98aa739d38e@app.fastmail.com>
 <8c71183a-0d28-4bcf-a806-78446ff95404@app.fastmail.com>
 <1009807.1760476747@sss.pgh.pa.us>
 <1F7227F5-C33D-4E2C-8511-33F1468590D0@gmail.com>
 <0a5a20d3-4621-46b3-b2ab-903f63a20dea@app.fastmail.com>
 <6F913129-ABEF-4004-AAF3-F!22!F!C!3!4!29AE8@gmail.com>
 <cbabbb45-ec27-47f1-a8f3-eb29073e9aef@app.fastmail.com>
 <ceab2c54-3463-4029-a7d8-895a617bb457@app.fastmail.com>
 <1547585.1760645808@sss.pgh.pa.us>
 <b8df8040-1b15-4a1a-b5f6-74acebbd2d0c@app.fastmail.com>
 <CAE7r3MLivh1sHWF06hrVXkiQbw-KChPcQsh+9CheXprm5vRVMQ@mail.gmail.com>
 <14865EB6-0BF4-462B-9072-> <422696.1768432145@sss.pgh.pa.us>
Subject: Re: Optimize LISTEN/NOTIFY
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/18521cfd-830e-4e8f-bcb0-eacee535480a%40app.fastmail.com>
Precedence: bulk

On Thu, Jan 15, 2026, at 00:09, Tom Lane wrote:
> I wrote:
>> I think that if we have a backend that isn't interested in our
>> notifications, and we can't direct-advance it, we should apply
>> the same behavior that was previously used for backends in other
>> databases.  That was basically a conservative approximation to
>> "isn't interested", and I don't see why it wouldn't work fine
>> when we have a more accurate idea of "isn't interested".
>
> I spent some time trying to measure the impact of that point,
> by modifying the test program you posted upthread so that
> some notifiers go at full speed while others respond to the
> rate-limit switch so that they can be made to go slowly.
> I couldn't really see any difference between what you have in v34
> and doing this the old way.  Maybe I just failed to construct the
> right test case to stress this behavior.  But anyway, without
> concrete evidence for a change I think we should stick to the
> old behavior.  When that was put in, it was to ameliorate a
> demonstrable "thundering herd" problem, and it seems likely to me
> that we'll bring that back for some usage patterns if we eagerly
> awaken backends that don't really need to do anything right away.

I reran the old benchmark [1] and got almost identical results as before
on my MacBook Pro M3 Max, when I tested v34 against patching v34 with
adding back the QUEUE_CLEANUP_DELAY logic:

```diff
@@ -2244,6 +2256,16 @@ SignalBackends(void)
                QUEUE_POS_PRECEDES(QUEUE_BACKEND_ADVANCING_POS(i), queueHeadAfterWrite) :
                QUEUE_POS_PRECEDES(pos, queueHeadBeforeWrite))
            {
+               /*
+                * This backend isn't interested in our notifications.  Rather
+                * than wake it repeatedly to skip over messages it doesn't
+                * care about, let the work accumulate so it can be done in
+                * larger, more efficient batches.
+                */
+               if (asyncQueuePageDiff(QUEUE_POS_PAGE(QUEUE_HEAD),
+                                      QUEUE_POS_PAGE(pos)) < QUEUE_CLEANUP_DELAY)
+                   continue;
+
                Assert(pid != InvalidPid);

                QUEUE_BACKEND_WAKEUP_PENDING(i) = true;
```

# MacBook Pro M3 Max

*** v34:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 155527 sent (15705/s), 155527 received (15705/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #          25110 (16.1%) avg: 0.080ms
 0.10-1.00ms     #####      78523 (50.5%) avg: 0.218ms
 1.00-10.00ms    ###        46798 (30.1%) avg: 4.315ms
 10.00-100.00ms  #          5096 (3.3%) avg: 13.002ms
>100.00ms                  0 (0.0%) avg: 0.000ms

*** v34 with QUEUE_CLEANUP_DELAY patch:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 44410 sent (5017/s), 44411 received (5018/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #          1289 (2.9%) avg: 0.082ms
 0.10-1.00ms     #          5588 (12.6%) avg: 0.262ms
 1.00-10.00ms    #          5155 (11.6%) avg: 4.807ms
 10.00-100.00ms  #####      23573 (53.1%) avg: 48.444ms
>100.00ms       #          8806 (19.8%) avg: 128.657ms

I repeated this three times and got very similar distributions and
averages.

However, I completely failed to reproduce this difference on my Intel
and AMD machines!

# AMD Ryzen 9 7950X3D 16-Core Processor:

*** v34:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 199123 sent (19948/s), 199123 received (19948/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #########  192536 (96.7%) avg: 0.050ms
 0.10-1.00ms     #          4577 (2.3%) avg: 0.264ms
 1.00-10.00ms    #          1806 (0.9%) avg: 3.304ms
 10.00-100.00ms  #          204 (0.1%) avg: 12.579ms
>100.00ms                  0 (0.0%) avg: 0.000ms

*** v34 with QUEUE_CLEANUP_DELAY patch:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 185687 sent (19267/s), 185686 received (19266/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #########  168090 (90.5%) avg: 0.052ms
 0.10-1.00ms     #          6112 (3.3%) avg: 0.194ms
 1.00-10.00ms    #          5818 (3.1%) avg: 5.823ms
 10.00-100.00ms  #          5666 (3.1%) avg: 15.340ms
>100.00ms                  0 (0.0%) avg: 0.000ms

# Intel(R) Core(TM) i9-14900K

*** v34:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 107836 sent (11367/s), 107836 received (11370/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #          130 (0.1%) avg: 0.060ms
 0.10-1.00ms     #          269 (0.2%) avg: 0.227ms
 1.00-10.00ms    ######     66886 (62.0%) avg: 8.115ms
 10.00-100.00ms  ###        40551 (37.6%) avg: 15.211ms
>100.00ms                  0 (0.0%) avg: 0.000ms

*** v34 with QUEUE_CLEANUP_DELAY patch:
./pg_async_notify_test --listeners 1 --notifiers 1 --channels 1000 --sleep 0.01 --sleep-exp 1.01
10 s: 97480 sent (10765/s), 97480 received (10764/s)
Notification Latency Distribution:
 0.00-0.01ms                0 (0.0%) avg: 0.000ms
 0.01-0.10ms     #          9980 (10.2%) avg: 0.065ms
 0.10-1.00ms     #          15703 (16.1%) avg: 0.352ms
 1.00-10.00ms    ####       47123 (48.3%) avg: 5.831ms
 10.00-100.00ms  ##         24674 (25.3%) avg: 44.263ms
>100.00ms                  0 (0.0%) avg: 0.000ms

I have no idea what could explain the difference on my M3 Max. Not sure
if it's due to macOS or due to the aarch64 CPU. It's still much faster
than master, so I think this is fine, we can always come back to this in
the future, if there is evidence this is not just an edge-case. Maybe
I'll do some benchmarking on a aarch64 server for fun, to see if it's
due to macOS or aarch64, or something else entirely.

I therefore agree with your change of bringing back the "wake laggers"
logic, even though it could possibly cause a few listening backends to
receive their notifications a bit later than they otherwise would.
Hopefully the spared context switches will allow a bit more throughput,
to make up for the increased delivery time of a few notifications.

> So, here's a reworked v35, which also incorporates quite a lot
> of cosmetic modifications as well as some bug fixes (mostly
> to do with being sure we recover from foreseeble problems like
> OOM partway through commit).  I think this is pretty close to
> committable, but if you see anything you don't like, let me know.

Nice improvements.  I agree with your decision of simplifying by getting
rid of advancingPos, and simply not touch advancing backends. I couldn't
measure any significant regression due to this simplification, very
nice.

Benchmarks on all my three machines look really good, and that benchmark
also does correctness testing, verifying that all notifications sent are
delivered, in order, without gaps.  All such tests pass.

/Joel

[1] https://www.postgresql.org/message-id/dab234b5-a10c-4fb5-a2b1-ce725d3e3020%40app.fastmail.com