From: Álvaro Herrera <alvherre@kurilemu.de>
To: Nick Ivanov <nick.ivanov@enterprisedb.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Possible race condition in pg_basebackup
Date: Fri, 21 Aug 2026 18:35:49 +0200
Message-ID: <aoh9KdCo12Y07mZO@alvherre.pgsql> (raw)
In-Reply-To: <CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com>
Hello Nick
On 2026-Aug-21, Nick Ivanov wrote:
> We are encountering a possible race condition when executing several
> `pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
> while initialising streaming replicas. Our automation starts 3
> pg_basebackup processes on 3 future replica servers within 1 second of each
> other. One of them almost always fails with "requested WAL segment ... has
> already been removed".
I think this is related to this thread here:
https://www.postgresql.org/message-id/flat/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj%40aliba...
and to this commit
Author: Amit Kapila <akapila@postgresql.org>
Branch: master Release: REL_19_BR [006dd4b2e] 2025-12-08 05:21:22 +0000
Branch: REL_18_STABLE Release: REL_18_2 [d3ceb2084] 2025-12-08 05:33:14 +0000
Prevent invalidation of newly created replication slots.
A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.
Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.
The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation to serialize WAL reservation and checkpoint's
minimum restart_lsn computation. This ensures that, if WAL reservation
occurs first, the checkpoint waits until restart_lsn is updated before
removing WAL. If the checkpoint runs first, subsequent WAL reservations
pick a position at or after the latest checkpoint's redo pointer.
We can't use the same fix for branch 17 and prior because commit
2090edc6f3 changed to compute to the minimum restart_LSN among slot's at
the beginning of checkpoint (or restart point). The fix for 17 and prior
branches is under discussion and will be committed separately.
Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
Author: Hou Zhijie <houzj.fnst@fujitsu.com>
Reviewed-by: Vitaly Davydov <v.davydov@postgrespro.ru>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 18
Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com
and to this other commit
Author: Amit Kapila <akapila@postgresql.org>
Branch: REL_17_STABLE Release: REL_17_8 [3510ebeb0] 2026-01-08 07:17:56 +0000
Branch: REL_16_STABLE Release: REL_16_12 [24cce33c3] 2026-01-08 07:07:23 +0000
Branch: REL_15_STABLE Release: REL_15_16 [aae05622a] 2026-01-08 06:54:52 +0000
Branch: REL_14_STABLE Release: REL_14_21 [7406df605] 2026-01-08 06:44:28 +0000
Prevent invalidation of newly created replication slots.
A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.
Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.
The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation, and a shared lock during the minimum LSN
calculation at checkpoints to serialize the process. This ensures that, if
WAL reservation occurs first, the checkpoint waits until restart_lsn is
updated before calculating the minimum LSN. If the checkpoint runs first,
subsequent WAL reservations pick a position at or after the latest
checkpoint's redo pointer.
We used a similar fix in HEAD (via commit 006dd4b2e5) and 18. The
difference is that in 17 and prior branches we need to additionally handle
the race condition with slot's minimum LSN computation during checkpoints.
Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
Author: Hou Zhijie <houzj.fnst@fujitsu.com>
Author: vignesh C <vignesh21@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 14
Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com
What version are you using?
If you're using a version that contains these fixes, then we may have
some slightly different bug ...
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: alvherre@kurilemu.de, nick.ivanov@enterprisedb.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Possible race condition in pg_basebackup
In-Reply-To: <aoh9KdCo12Y07mZO@alvherre.pgsql>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox