Feedback-ID: ie3de48e3:Fastmail
Date: Fri, 21 Aug 2026 18:35:49 +0200
From: =?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de>
To: Nick Ivanov <nick.ivanov@enterprisedb.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Possible race condition in pg_basebackup
Message-ID: <aoh9KdCo12Y07mZO@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/aoh9KdCo12Y07mZO%40alvherre.pgsql>
Precedence: bulk

Hello Nick

On 2026-Aug-21, Nick Ivanov wrote:

> We are encountering a possible race condition when executing several
> `pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
> while initialising streaming replicas. Our automation starts 3
> pg_basebackup processes on 3 future replica servers within 1 second of each
> other. One of them almost always fails with "requested WAL segment ... has
> already been removed".

I think this is related to this thread here:
https://www.postgresql.org/message-id/flat/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj%40alibaba-inc.com

and to this commit

Author: Amit Kapila <akapila@postgresql.org>
Branch: master Release: REL_19_BR [006dd4b2e] 2025-12-08 05:21:22 +0000
Branch: REL_18_STABLE Release: REL_18_2 [d3ceb2084] 2025-12-08 05:33:14 +0000

    Prevent invalidation of newly created replication slots.
    
    A race condition could cause a newly created replication slot to become
    invalidated between WAL reservation and a checkpoint.
    
    Previously, if the required WAL was removed, we retried the reservation
    process. However, the slot could still be invalidated before the retry if
    the WAL was not yet removed but the checkpoint advanced the redo pointer
    beyond the slot's intended restart LSN and computed the minimum LSN that
    needs to be preserved for the slots.
    
    The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
    during WAL reservation to serialize WAL reservation and checkpoint's
    minimum restart_lsn computation. This ensures that, if WAL reservation
    occurs first, the checkpoint waits until restart_lsn is updated before
    removing WAL. If the checkpoint runs first, subsequent WAL reservations
    pick a position at or after the latest checkpoint's redo pointer.
    
    We can't use the same fix for branch 17 and prior because commit
    2090edc6f3 changed to compute to the minimum restart_LSN among slot's at
    the beginning of checkpoint (or restart point). The fix for 17 and prior
    branches is under discussion and will be committed separately.
    
    Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
    Author: Hou Zhijie <houzj.fnst@fujitsu.com>
    Reviewed-by: Vitaly Davydov <v.davydov@postgrespro.ru>
    Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
    Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
    Backpatch-through: 18
    Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com


and to this other commit


Author: Amit Kapila <akapila@postgresql.org>
Branch: REL_17_STABLE Release: REL_17_8 [3510ebeb0] 2026-01-08 07:17:56 +0000
Branch: REL_16_STABLE Release: REL_16_12 [24cce33c3] 2026-01-08 07:07:23 +0000
Branch: REL_15_STABLE Release: REL_15_16 [aae05622a] 2026-01-08 06:54:52 +0000
Branch: REL_14_STABLE Release: REL_14_21 [7406df605] 2026-01-08 06:44:28 +0000

    Prevent invalidation of newly created replication slots.
    
    A race condition could cause a newly created replication slot to become
    invalidated between WAL reservation and a checkpoint.
    
    Previously, if the required WAL was removed, we retried the reservation
    process. However, the slot could still be invalidated before the retry if
    the WAL was not yet removed but the checkpoint advanced the redo pointer
    beyond the slot's intended restart LSN and computed the minimum LSN that
    needs to be preserved for the slots.
    
    The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
    during WAL reservation, and a shared lock during the minimum LSN
    calculation at checkpoints to serialize the process. This ensures that, if
    WAL reservation occurs first, the checkpoint waits until restart_lsn is
    updated before calculating the minimum LSN. If the checkpoint runs first,
    subsequent WAL reservations pick a position at or after the latest
    checkpoint's redo pointer.
    
    We used a similar fix in HEAD (via commit 006dd4b2e5) and 18. The
    difference is that in 17 and prior branches we need to additionally handle
    the race condition with slot's minimum LSN computation during checkpoints.
    
    Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
    Author: Hou Zhijie <houzj.fnst@fujitsu.com>
    Author: vignesh C <vignesh21@gmail.com>
    Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
    Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
    Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
    Backpatch-through: 14
    Discussion: https://postgr.es/m/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com


What version are you using?

If you're using a version that contains these fixes, then we may have
some slightly different bug ...

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)