Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJZvK-00000000JD3-2eMJ for pgsql-bugs@arkaria.postgresql.org; Wed, 21 Oct 2026 17:16:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xJZvK-00000006KEf-0STs for pgsql-bugs@arkaria.postgresql.org; Wed, 21 Oct 2026 17:16:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJZvJ-00000006KEW-3cAi for pgsql-bugs@lists.postgresql.org; Wed, 21 Oct 2026 17:16:21 +0000 Received: from mail.zonezero.se ([46.246.30.157]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJZvH-00000000GXK-02sA for pgsql-bugs@lists.postgresql.org; Wed, 21 Oct 2026 17:16:21 +0000 Received: from mail.zonezero.se (unknown [10.100.2.1]) by mail.zonezero.se (Postfix) with ESMTPS id B07D066A; Wed, 21 Oct 2026 19:16:18 +0200 (CEST) Received: from webmail.zonezero.se (localhost.localdomain [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.zonezero.se (Postfix) with ESMTPSA id 697661203BA; Wed, 21 Oct 2026 19:16:18 +0200 (CEST) Authentication-Results: ORIGINATING; auth=pass smtp.auth=admin@zonezero.se smtp.mailfrom=jesper@stjohn.se MIME-Version: 1.0 Date: Wed, 21 Oct 2026 19:16:18 +0200 From: Jesper St John To: Kirill Reshke Cc: pgsql-bugs@lists.postgresql.org Subject: Re: Timing issue in pg_basebackup In-Reply-To: References: Message-ID: <4417791388e8f1df20f64013e944be8e@stjohn.se> X-Sender: jesper@stjohn.se Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2026-10-20 23:11, Kirill Reshke wrote: > On Mon, 19 Oct 2026 at 19:33, Jesper St John wrote: >> >> Hey! >> >> I recently came across this issue when running pg_basebackup from 2 >> nodes at the same time through ansible. One of the nodes sometimes >> reported: "requested WAL segment ... has already been removed". Tested >> on v16, v17 and v18. The workaround was running them in sequence or >> pause between runs. >> >> It looks like there's a timing issue here: >> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2. >> 2. Backup A hasn't reserved that WAL yet. >> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) >> - >> deletes segment 2. >> 4. Backup A asks for segment 2, which is gone, and fails. > > Hi! > Looks like this is a real issue. Thanks for looking into it. > >> I've included a patch that moves the replication slot handing to run >> before the backup part as a separate function. >> >> Some possible caveats already: >> * Since moving the slot creation, we might run in to trouble if for >> example, --checkpoint=spread is used since the slot is open for >> longer. >> * If any error arises in the backup part, now there will be a >> lingering >> slot when previously no slot was created. > > lingering slots looks like a real regression here. can we remove them > when basebackup client disconnects? A lingering slot could also happen today - only with this patch there's a higher risk of it happening. Would some cleanup code in a function and use it with atexit() work? > > NIT: comment in PrepareLogStreamer loses its meaning after code motion: > > + /* Get a second connection */ > + bgconn = GetConnection(); > I've moved the code to its own function - the comment were there already. > >> *** DISCLAIMER *** >> I've used an LLM to help me write the test - I'm rubbish at Perl. > > OK