pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Timing issue in pg_basebackup
11+ messages / 7 participants
[nested] [flat]

* Timing issue in pg_basebackup
@ 2026-10-19 14:33  Jesper St John <jesper@stjohn.se>
  0 siblings, 1 reply; 11+ messages in thread

From: Jesper St John @ 2026-10-19 14:33 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org

Hey!

I recently came across this issue when running pg_basebackup from 2 
nodes at the same time through ansible. One of the nodes sometimes 
reported: "requested WAL segment ... has already been removed". Tested 
on v16, v17 and v18. The workaround was running them in sequence or 
pause between runs.

It looks like there's a timing issue here:
1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
2. Backup A hasn't reserved that WAL yet.
3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) - 
deletes segment 2.
4. Backup A asks for segment 2, which is gone, and fails.

I've included a patch that moves the replication slot handing to run 
before the backup part as a separate function.

Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for 
example, --checkpoint=spread is used since the slot is open for longer.
* If any error arises in the backup part, now there will be a lingering 
slot when previously no slot was created.

*** DISCLAIMER ***
I've used an LLM to help me write the test - I'm rubbish at Perl.

Love to get some feedback.

Regards,
Jesper St John

Attachments:

v1-0001-pg_basebackup-Create-the-WAL-streaming-slot-befor.patch (text/x-diff, 9.9K) download From de8268b69969531952ca411b5c1490b4bb660b7f Mon Sep 17 00:00:00 2001 From: Jesper St John <jesper@stjohn.se> Date: Mon, 19 Oct 2026 15:35:59 +0200 Subject: [PATCH v1] pg_basebackup: Create the WAL streaming slot before starting the backup With --wal-method=stream, pg_basebackup created its replication slot only after BASE_BACKUP had returned the backup's start position. A checkpoint running in between could remove WAL from that position onwards, and the backup then failed with "requested WAL segment ... has already been removed". Open the WAL streaming connection and create its slot before sending BASE_BACKUP, rather than after it has returned the start position, so the slot reserves all the WAL the backup needs. Add a TAP test that uses an injection point to hold slot creation while WAL is advanced and a checkpoint is run. --- src/bin/pg_basebackup/Makefile | 4 +- src/bin/pg_basebackup/meson.build | 2 + src/bin/pg_basebackup/pg_basebackup.c | 96 ++++++++++++------- .../pg_basebackup/t/012_wal_slot_timing.pl | 82 ++++++++++++++++ 4 files changed, 149 insertions(+), 35 deletions(-) create mode 100644 src/bin/pg_basebackup/t/012_wal_slot_timing.pl diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile index df94fc27d02..12ff32e329b 100644 --- a/src/bin/pg_basebackup/Makefile +++ b/src/bin/pg_basebackup/Makefile @@ -12,13 +12,15 @@ PGFILEDESC = "pg_basebackup/pg_receivewal/pg_recvlogical - streaming WAL and backup receivers" PGAPPICON=win32 -EXTRA_INSTALL=contrib/test_decoding +EXTRA_INSTALL=contrib/test_decoding \ + src/test/modules/injection_points subdir = src/bin/pg_basebackup top_builddir = ../../.. include $(top_builddir)/src/Makefile.global # make these available to TAP test scripts +export enable_injection_points export LZ4 export TAR # Note that GZIP cannot be used directly as this environment variable is diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build index d70ce5786a2..c10a56512a0 100644 --- a/src/bin/pg_basebackup/meson.build +++ b/src/bin/pg_basebackup/meson.build @@ -96,10 +96,12 @@ tests += { 'env': {'GZIP_PROGRAM': gzip.found() ? gzip.full_path() : '', 'TAR': tar.found() ? tar.full_path() : '', 'LZ4': program_lz4.found() ? program_lz4.full_path() : '', + 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no', }, 'tests': [ 't/010_pg_basebackup.pl', 't/011_in_place_tablespace.pl', + 't/012_wal_slot_timing.pl', 't/020_pg_receivewal.pl', 't/030_pg_recvlogical.pl', 't/040_pg_createsubscriber.pl', diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index c3b87a19e76..e48e75ddd6d 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -165,6 +165,9 @@ static bool found_existing_xlogdir = false; static bool made_tablespace_dirs = false; static bool found_tablespace_dirs = false; +/* Connection for the background WAL receiver, see PrepareLogStreamer() */ +static PGconn *bgconn = NULL; + /* Progress indicators */ static uint64 totalsize_kb; static uint64 totaldone; @@ -607,6 +610,55 @@ LogStreamerMain(logstreamer_param *param) return 0; } +/* + * Open the connection for the background WAL receiver and create its + * replication slot, if one is needed. + * + * This must be done before BASE_BACKUP is sent: the slot reserves WAL from + * its creation onwards, so creating it first guarantees that no WAL from the + * backup's start position onwards can be removed before the receiver starts + * streaming. If the slot were created only after BASE_BACKUP had returned + * the start position, a checkpoint in between could remove WAL the backup + * needs. + */ +static void +PrepareLogStreamer(void) +{ + /* Get a second connection */ + bgconn = GetConnection(); + if (!bgconn) + /* Error message already written in GetConnection() */ + exit(1); + + /* Temporary replication slots are only supported in 10 and newer */ + if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) + temp_replication_slot = false; + + /* + * Create replication slot if requested + */ + if (temp_replication_slot && !replication_slot) + replication_slot = psprintf("pg_basebackup_%u", + (unsigned int) PQbackendPID(bgconn)); + if (temp_replication_slot || create_slot) + { + if (!CreateReplicationSlot(bgconn, replication_slot, NULL, + temp_replication_slot, true, true, false, + false, false)) + exit(1); + + if (verbose) + { + if (temp_replication_slot) + pg_log_info("created temporary replication slot \"%s\"", + replication_slot); + else + pg_log_info("created replication slot \"%s\"", + replication_slot); + } + } +} + /* * Initiate background process for receiving xlog during the backup. * The background stream will use its own database connection so we can @@ -620,7 +672,10 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, logstreamer_param *param; char statusdir[MAXPGPATH]; + Assert(bgconn != NULL); + param = pg_malloc0_object(logstreamer_param); + param->bgconn = bgconn; param->timeline = timeline; param->sysidentifier = sysidentifier; param->wal_compress_algorithm = wal_compress_algorithm; @@ -639,46 +694,12 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, pg_fatal("could not create pipe for background process: %m"); #endif - /* Get a second connection */ - param->bgconn = GetConnection(); - if (!param->bgconn) - /* Error message already written in GetConnection() */ - exit(1); - /* In post-10 cluster, pg_xlog has been renamed to pg_wal */ snprintf(param->xlog, sizeof(param->xlog), "%s/%s", basedir, PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ? "pg_xlog" : "pg_wal"); - /* Temporary replication slots are only supported in 10 and newer */ - if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) - temp_replication_slot = false; - - /* - * Create replication slot if requested - */ - if (temp_replication_slot && !replication_slot) - replication_slot = psprintf("pg_basebackup_%u", - (unsigned int) PQbackendPID(param->bgconn)); - if (temp_replication_slot || create_slot) - { - if (!CreateReplicationSlot(param->bgconn, replication_slot, NULL, - temp_replication_slot, true, true, false, - false, false)) - exit(1); - - if (verbose) - { - if (temp_replication_slot) - pg_log_info("created temporary replication slot \"%s\"", - replication_slot); - else - pg_log_info("created replication slot \"%s\"", - replication_slot); - } - } - if (format == 'p') { /* @@ -1970,6 +1991,13 @@ BaseBackup(char *compression_algorithm, char *compression_detail, compression_detail); } + /* + * If streaming WAL, create the replication slot before starting the + * backup, so that it protects all the WAL the backup needs. + */ + if (includewal == STREAM_WAL) + PrepareLogStreamer(); + if (verbose) pg_log_info("initiating base backup, waiting for checkpoint to complete"); diff --git a/src/bin/pg_basebackup/t/012_wal_slot_timing.pl b/src/bin/pg_basebackup/t/012_wal_slot_timing.pl new file mode 100644 index 00000000000..0de5e24058b --- /dev/null +++ b/src/bin/pg_basebackup/t/012_wal_slot_timing.pl @@ -0,0 +1,82 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Check that pg_basebackup's replication slot reserves WAL from before the +# backup's start position. If the slot were created only after BASE_BACKUP +# had returned the start position, a checkpoint running in between could +# remove the WAL the backup needs, and streaming it would fail. +# +# We hold the slot creation with an injection point, force WAL past several +# segments and checkpoint, then let the slot creation continue. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +if ($ENV{enable_injection_points} ne 'yes') +{ + plan skip_all => 'Injection points not supported by this build'; +} + +my $tempdir = PostgreSQL::Test::Utils::tempdir; + +my $node = PostgreSQL::Test::Cluster->new('main'); +$node->init(allows_streaming => 1); +$node->append_conf( + 'postgresql.conf', q{ +wal_keep_size = 0 +checkpoint_timeout = 1h +}); +$node->start; + +if (!$node->check_extension('injection_points')) +{ + plan skip_all => 'Extension injection_points not installed'; +} + +$node->safe_psql('postgres', 'CREATE EXTENSION injection_points'); +$node->safe_psql('postgres', + "SELECT injection_points_attach('replication-slot-create-begin', 'wait')" +); + +my $backupdir = "$tempdir/backup"; +my ($stdout, $stderr); +my $h = IPC::Run::start( + [ + 'pg_basebackup', '--no-sync', + '--checkpoint' => 'fast', + '--pgdata' => $backupdir, + '--wal-method' => 'stream', + '--dbname' => $node->connstr('postgres'), + ], + '>' => \$stdout, + '2>' => \$stderr); + +# Wait until the walsender is about to create pg_basebackup's slot. +$node->wait_for_event('walsender', 'replication-slot-create-begin'); + +# Move WAL forward by several segments and checkpoint, so that every segment +# not protected by a slot or wal_keep_size is removed or recycled. +$node->advance_wal(3); +$node->safe_psql('postgres', 'CHECKPOINT'); + +$node->safe_psql('postgres', + "SELECT injection_points_detach('replication-slot-create-begin')"); +$node->safe_psql('postgres', + "SELECT injection_points_wakeup('replication-slot-create-begin')"); + +my $ok = $h->finish; +ok($ok, 'pg_basebackup succeeds with WAL activity during slot creation') + or diag("pg_basebackup stderr:\n$stderr"); +unlike( + $stderr, + qr/has already been removed/, + 'no WAL needed by the backup was removed'); + +# pg_verifybackup also parses the WAL the backup needs, so this fails if +# any of it is missing. +$node->command_ok([ 'pg_verifybackup', $backupdir ], + 'backup with streamed WAL verifies'); + +done_testing(); -- 2.39.5
^ permalink raw reply [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-10-20 21:11  Kirill Reshke <reshkekirill@gmail.com>
  parent: Jesper St John <jesper@stjohn.se>
  0 siblings, 1 reply; 11+ messages in thread

From: Kirill Reshke @ 2026-10-20 21:11 UTC (permalink / raw)
  To: Jesper St John <jesper@stjohn.se>; +Cc: pgsql-bugs@lists.postgresql.org

On Mon, 19 Oct 2026 at 19:33, Jesper St John <jesper@stjohn.se> wrote:
>
> Hey!
>
> I recently came across this issue when running pg_basebackup from 2
> nodes at the same time through ansible. One of the nodes sometimes
> reported: "requested WAL segment ... has already been removed". Tested
> on v16, v17 and v18. The workaround was running them in sequence or
> pause between runs.
>
> It looks like there's a timing issue here:
> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
> 2. Backup A hasn't reserved that WAL yet.
> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) -
> deletes segment 2.
> 4. Backup A asks for segment 2, which is gone, and fails.

Hi!
Looks like this is a real issue.

> I've included a patch that moves the replication slot handing to run
> before the backup part as a separate function.
>
> Some possible caveats already:
> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.
> * If any error arises in the backup part, now there will be a lingering
> slot when previously no slot was created.

lingering slots looks like a real regression here. can we remove them
when basebackup client disconnects?

NIT: comment in PrepareLogStreamer loses its meaning after code motion:

+ /* Get a second connection */
+ bgconn = GetConnection();


> *** DISCLAIMER ***
> I've used an LLM to help me write the test - I'm rubbish at Perl.

OK



-- 
Best regards,
Kirill Reshke






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-10-21 17:16  Jesper St John <jesper@stjohn.se>
  parent: Kirill Reshke <reshkekirill@gmail.com>
  0 siblings, 1 reply; 11+ messages in thread

From: Jesper St John @ 2026-10-21 17:16 UTC (permalink / raw)
  To: Kirill Reshke <reshkekirill@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org

On 2026-10-20 23:11, Kirill Reshke wrote:
> On Mon, 19 Oct 2026 at 19:33, Jesper St John <jesper@stjohn.se> wrote:
>> 
>> Hey!
>> 
>> I recently came across this issue when running pg_basebackup from 2
>> nodes at the same time through ansible. One of the nodes sometimes
>> reported: "requested WAL segment ... has already been removed". Tested
>> on v16, v17 and v18. The workaround was running them in sequence or
>> pause between runs.
>> 
>> It looks like there's a timing issue here:
>> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
>> 2. Backup A hasn't reserved that WAL yet.
>> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) 
>> -
>> deletes segment 2.
>> 4. Backup A asks for segment 2, which is gone, and fails.
> 
> Hi!
> Looks like this is a real issue.

Thanks for looking into it.

> 
>> I've included a patch that moves the replication slot handing to run
>> before the backup part as a separate function.
>> 
>> Some possible caveats already:
>> * Since moving the slot creation, we might run in to trouble if for
>> example, --checkpoint=spread is used since the slot is open for 
>> longer.
>> * If any error arises in the backup part, now there will be a 
>> lingering
>> slot when previously no slot was created.
> 
> lingering slots looks like a real regression here. can we remove them
> when basebackup client disconnects?

A lingering slot could also happen today - only with this patch there's 
a higher risk of it happening.

Would some cleanup code in a function and use it with atexit() work?

> 
> NIT: comment in PrepareLogStreamer loses its meaning after code motion:
> 
> + /* Get a second connection */
> + bgconn = GetConnection();
> 

I've moved the code to its own function - the comment were there 
already.


> 
>> *** DISCLAIMER ***
>> I've used an LLM to help me write the test - I'm rubbish at Perl.
> 
> OK






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-10-22 05:44  Kirill Reshke <reshkekirill@gmail.com>
  parent: Jesper St John <jesper@stjohn.se>
  0 siblings, 2 replies; 11+ messages in thread

From: Kirill Reshke @ 2026-10-22 05:44 UTC (permalink / raw)
  To: Jesper St John <jesper@stjohn.se>; +Cc: pgsql-bugs@lists.postgresql.org

On Wed, 21 Oct 2026 at 22:16, Jesper St John <jesper@stjohn.se> wrote:
>
> On 2026-10-20 23:11, Kirill Reshke wrote:

> >
> >> I've included a patch that moves the replication slot handing to run
> >> before the backup part as a separate function.
> >>
> >> Some possible caveats already:
> >> * Since moving the slot creation, we might run in to trouble if for
> >> example, --checkpoint=spread is used since the slot is open for
> >> longer.
> >> * If any error arises in the backup part, now there will be a
> >> lingering
> >> slot when previously no slot was created.
> >
> > lingering slots looks like a real regression here. can we remove them
> > when basebackup client disconnects?
>
> A lingering slot could also happen today - only with this patch there's
> a higher risk of it happening.

Higher risk of lingering slots is also a regression. But looks like
this risk is anyway close to zero.

> Would some cleanup code in a function and use it with atexit() work?

Something like this should do, can be made a separate patch thought

> >
> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
> >
> > + /* Get a second connection */
> > + bgconn = GetConnection();
> >
>
> I've moved the code to its own function - the comment were there
> already.
>

Yep, the point is It looks very out-of-context here. Like, the first
line of the function says we're taking a second connection. Second to
what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
connection' or something.



-- 
Best regards,
Kirill Reshke






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-10-29 19:26  Jesper St John <jesper@stjohn.se>
  parent: Kirill Reshke <reshkekirill@gmail.com>
  1 sibling, 0 replies; 11+ messages in thread

From: Jesper St John @ 2026-10-29 19:26 UTC (permalink / raw)
  To: Kirill Reshke <reshkekirill@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org

Hi!

Here are 2 v2-patches as per your suggestion.

I'm a bit unsure if the check in 
drop_lingering_replication_slot_atexit() function is enough to not break 
anything.
Or too much.

Suggestions welcome, could also make it into 1 patch if you prefer that.

//Jesper

On 2026-10-22 07:44, Kirill Reshke wrote:
> On Wed, 21 Oct 2026 at 22:16, Jesper St John <jesper@stjohn.se> wrote:
>> 
>> On 2026-10-20 23:11, Kirill Reshke wrote:
> 
>> >
>> >> I've included a patch that moves the replication slot handing to run
>> >> before the backup part as a separate function.
>> >>
>> >> Some possible caveats already:
>> >> * Since moving the slot creation, we might run in to trouble if for
>> >> example, --checkpoint=spread is used since the slot is open for
>> >> longer.
>> >> * If any error arises in the backup part, now there will be a
>> >> lingering
>> >> slot when previously no slot was created.
>> >
>> > lingering slots looks like a real regression here. can we remove them
>> > when basebackup client disconnects?
>> 
>> A lingering slot could also happen today - only with this patch 
>> there's
>> a higher risk of it happening.
> 
> Higher risk of lingering slots is also a regression. But looks like
> this risk is anyway close to zero.
> 
>> Would some cleanup code in a function and use it with atexit() work?
> 
> Something like this should do, can be made a separate patch thought
> 
>> >
>> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
>> >
>> > + /* Get a second connection */
>> > + bgconn = GetConnection();
>> >
>> 
>> I've moved the code to its own function - the comment were there
>> already.
>> 
> 
> Yep, the point is It looks very out-of-context here. Like, the first
> line of the function says we're taking a second connection. Second to
> what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
> connection' or something.

Attachments: download all (2 files, .tar.gz)

v2-0002-pg_basebackup-Drop-a-lingering-slot-if-the-backup.patch (text/x-diff, 3.9K) download From a66192994b42d09a2a4134b5e655d4fa7c15aa61 Mon Sep 17 00:00:00 2001 From: Jesper St John <jesper@stjohn.se> Date: Wed, 28 Oct 2026 11:22:27 +0100 Subject: [PATCH v2 2/2] pg_basebackup: Drop a lingering slot if the backup fails to start Register an atexit handler, once the slot has been created, that drops it if pg_basebackup exits before the background WAL receiver has taken over the connection. Add tests to 010_pg_basebackup.pl that use a bogus --manifest-checksums value to fail BASE_BACKUP after the slot has been created, and check that only the new slot is dropped. --- src/bin/pg_basebackup/pg_basebackup.c | 23 +++++++++++ src/bin/pg_basebackup/t/010_pg_basebackup.pl | 42 ++++++++++++++++++++ 2 files changed, 65 insertions(+) diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index 18e36451290..86144f01aa4 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -317,6 +317,22 @@ kill_bgchild_atexit(void) } #endif +/* + * Drop the lingering slot created by PrepareLogStreamer() if we fail before the + * background WAL receiver has taken over its connection. Only registered + * once the slot exists, so a pre-existing slot of the same name should never + * be dropped. + */ +static void +drop_lingering_replication_slot_atexit(void) +{ + /* Check that we have a good state */ + if (success || in_log_streamer || bgchild > 0) + return; + + (void) DropReplicationSlot(bgconn, replication_slot); +} + /* * Split argument into old_dir and new_dir and append to tablespace mapping * list. @@ -647,6 +663,13 @@ PrepareLogStreamer(void) false, false)) exit(1); + /* + * Only register atexit function if create_slot is true. Temporary + * slots go away with connection + */ + if (create_slot) + atexit(drop_lingering_replication_slot_atexit); + if (verbose) { if (temp_replication_slot) diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index 2442131e179..b6a31703262 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -828,9 +828,51 @@ $node->command_fails_like( qr/replication slot "slot0" already exists/, 'pg_basebackup fails with --create-slot --slot and a previously existing slot' ); +is( $node->safe_psql( + 'postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'slot0'} + ), + 'slot0', + 'previously existing slot is not dropped when --create-slot fails'); + +# The slot is created before BASE_BACKUP is sent, so a failure of +# BASE_BACKUP itself must not leave a slot created by --create-slot behind. +$node->command_fails_like( + [ + @pg_basebackup_defs, + '--pgdata' => "$tempdir/backupxs_slot2", + '--create-slot', + '--slot' => 'slot2', + '--manifest-checksums' => 'bogus' + ], + qr/unrecognized checksum algorithm/, + 'pg_basebackup --create-slot fails when BASE_BACKUP fails'); +is( $node->safe_psql( + 'postgres', + q{SELECT count(*) FROM pg_replication_slots WHERE slot_name = 'slot2'} + ), + '0', + 'slot created by failed pg_basebackup --create-slot is dropped'); $node->safe_psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')}); + +# Without --create-slot, a failed backup must leave the given slot alone. +$node->command_fails_like( + [ + @pg_basebackup_defs, + '--pgdata' => "$tempdir/backupxs_slot1_fail", + '--slot' => 'slot1', + '--manifest-checksums' => 'bogus' + ], + qr/unrecognized checksum algorithm/, + 'pg_basebackup --slot fails when BASE_BACKUP fails'); +is( $node->safe_psql( + 'postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'slot1'} + ), + 'slot1', + 'existing slot is not dropped when pg_basebackup --slot fails'); my $lsn = $node->safe_psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'} ); -- 2.56.0
v2-0001-pg_basebackup-Create-the-WAL-streaming-slot-befor.patch (text/x-diff, 9.9K) download From c7ddb97cbd7f923cd9407b1d51313f86fbf587de Mon Sep 17 00:00:00 2001 From: Jesper St John <jesper@stjohn.se> Date: Mon, 19 Oct 2026 15:35:59 +0200 Subject: [PATCH v2 1/2] pg_basebackup: Create the WAL streaming slot before starting the backup With --wal-method=stream, pg_basebackup created its replication slot only after BASE_BACKUP had returned the backup's start position. A checkpoint running in between could remove WAL from that position onwards, and the backup then failed with "requested WAL segment ... has already been removed". Open the WAL streaming connection and create its slot before sending BASE_BACKUP, rather than after it has returned the start position, so the slot reserves all the WAL the backup needs. Add a TAP test that uses an injection point to hold slot creation while WAL is advanced and a checkpoint is run. --- src/bin/pg_basebackup/Makefile | 4 +- src/bin/pg_basebackup/meson.build | 2 + src/bin/pg_basebackup/pg_basebackup.c | 96 ++++++++++++------- .../pg_basebackup/t/012_wal_slot_timing.pl | 82 ++++++++++++++++ 4 files changed, 149 insertions(+), 35 deletions(-) create mode 100644 src/bin/pg_basebackup/t/012_wal_slot_timing.pl diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile index df94fc27d02..12ff32e329b 100644 --- a/src/bin/pg_basebackup/Makefile +++ b/src/bin/pg_basebackup/Makefile @@ -12,13 +12,15 @@ PGFILEDESC = "pg_basebackup/pg_receivewal/pg_recvlogical - streaming WAL and backup receivers" PGAPPICON=win32 -EXTRA_INSTALL=contrib/test_decoding +EXTRA_INSTALL=contrib/test_decoding \ + src/test/modules/injection_points subdir = src/bin/pg_basebackup top_builddir = ../../.. include $(top_builddir)/src/Makefile.global # make these available to TAP test scripts +export enable_injection_points export LZ4 export TAR # Note that GZIP cannot be used directly as this environment variable is diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build index d70ce5786a2..c10a56512a0 100644 --- a/src/bin/pg_basebackup/meson.build +++ b/src/bin/pg_basebackup/meson.build @@ -96,10 +96,12 @@ tests += { 'env': {'GZIP_PROGRAM': gzip.found() ? gzip.full_path() : '', 'TAR': tar.found() ? tar.full_path() : '', 'LZ4': program_lz4.found() ? program_lz4.full_path() : '', + 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no', }, 'tests': [ 't/010_pg_basebackup.pl', 't/011_in_place_tablespace.pl', + 't/012_wal_slot_timing.pl', 't/020_pg_receivewal.pl', 't/030_pg_recvlogical.pl', 't/040_pg_createsubscriber.pl', diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index c3b87a19e76..18e36451290 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -165,6 +165,9 @@ static bool found_existing_xlogdir = false; static bool made_tablespace_dirs = false; static bool found_tablespace_dirs = false; +/* Connection for the background WAL receiver, see PrepareLogStreamer() */ +static PGconn *bgconn = NULL; + /* Progress indicators */ static uint64 totalsize_kb; static uint64 totaldone; @@ -607,6 +610,55 @@ LogStreamerMain(logstreamer_param *param) return 0; } +/* + * Open the connection for the background WAL receiver and create its + * replication slot, if one is needed. + * + * This must be done before BASE_BACKUP is sent: the slot reserves WAL from + * its creation onwards, so creating it first guarantees that no WAL from the + * backup's start position onwards can be removed before the receiver starts + * streaming. If the slot were created only after BASE_BACKUP had returned + * the start position, a checkpoint in between could remove WAL the backup + * needs. + */ +static void +PrepareLogStreamer(void) +{ + /* Get a WAL serving connection */ + bgconn = GetConnection(); + if (!bgconn) + /* Error message already written in GetConnection() */ + exit(1); + + /* Temporary replication slots are only supported in 10 and newer */ + if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) + temp_replication_slot = false; + + /* + * Create replication slot if requested + */ + if (temp_replication_slot && !replication_slot) + replication_slot = psprintf("pg_basebackup_%u", + (unsigned int) PQbackendPID(bgconn)); + if (temp_replication_slot || create_slot) + { + if (!CreateReplicationSlot(bgconn, replication_slot, NULL, + temp_replication_slot, true, true, false, + false, false)) + exit(1); + + if (verbose) + { + if (temp_replication_slot) + pg_log_info("created temporary replication slot \"%s\"", + replication_slot); + else + pg_log_info("created replication slot \"%s\"", + replication_slot); + } + } +} + /* * Initiate background process for receiving xlog during the backup. * The background stream will use its own database connection so we can @@ -620,7 +672,10 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, logstreamer_param *param; char statusdir[MAXPGPATH]; + Assert(bgconn != NULL); + param = pg_malloc0_object(logstreamer_param); + param->bgconn = bgconn; param->timeline = timeline; param->sysidentifier = sysidentifier; param->wal_compress_algorithm = wal_compress_algorithm; @@ -639,46 +694,12 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, pg_fatal("could not create pipe for background process: %m"); #endif - /* Get a second connection */ - param->bgconn = GetConnection(); - if (!param->bgconn) - /* Error message already written in GetConnection() */ - exit(1); - /* In post-10 cluster, pg_xlog has been renamed to pg_wal */ snprintf(param->xlog, sizeof(param->xlog), "%s/%s", basedir, PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ? "pg_xlog" : "pg_wal"); - /* Temporary replication slots are only supported in 10 and newer */ - if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) - temp_replication_slot = false; - - /* - * Create replication slot if requested - */ - if (temp_replication_slot && !replication_slot) - replication_slot = psprintf("pg_basebackup_%u", - (unsigned int) PQbackendPID(param->bgconn)); - if (temp_replication_slot || create_slot) - { - if (!CreateReplicationSlot(param->bgconn, replication_slot, NULL, - temp_replication_slot, true, true, false, - false, false)) - exit(1); - - if (verbose) - { - if (temp_replication_slot) - pg_log_info("created temporary replication slot \"%s\"", - replication_slot); - else - pg_log_info("created replication slot \"%s\"", - replication_slot); - } - } - if (format == 'p') { /* @@ -1970,6 +1991,13 @@ BaseBackup(char *compression_algorithm, char *compression_detail, compression_detail); } + /* + * If streaming WAL, create the replication slot before starting the + * backup, so that it protects all the WAL the backup needs. + */ + if (includewal == STREAM_WAL) + PrepareLogStreamer(); + if (verbose) pg_log_info("initiating base backup, waiting for checkpoint to complete"); diff --git a/src/bin/pg_basebackup/t/012_wal_slot_timing.pl b/src/bin/pg_basebackup/t/012_wal_slot_timing.pl new file mode 100644 index 00000000000..0de5e24058b --- /dev/null +++ b/src/bin/pg_basebackup/t/012_wal_slot_timing.pl @@ -0,0 +1,82 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Check that pg_basebackup's replication slot reserves WAL from before the +# backup's start position. If the slot were created only after BASE_BACKUP +# had returned the start position, a checkpoint running in between could +# remove the WAL the backup needs, and streaming it would fail. +# +# We hold the slot creation with an injection point, force WAL past several +# segments and checkpoint, then let the slot creation continue. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +if ($ENV{enable_injection_points} ne 'yes') +{ + plan skip_all => 'Injection points not supported by this build'; +} + +my $tempdir = PostgreSQL::Test::Utils::tempdir; + +my $node = PostgreSQL::Test::Cluster->new('main'); +$node->init(allows_streaming => 1); +$node->append_conf( + 'postgresql.conf', q{ +wal_keep_size = 0 +checkpoint_timeout = 1h +}); +$node->start; + +if (!$node->check_extension('injection_points')) +{ + plan skip_all => 'Extension injection_points not installed'; +} + +$node->safe_psql('postgres', 'CREATE EXTENSION injection_points'); +$node->safe_psql('postgres', + "SELECT injection_points_attach('replication-slot-create-begin', 'wait')" +); + +my $backupdir = "$tempdir/backup"; +my ($stdout, $stderr); +my $h = IPC::Run::start( + [ + 'pg_basebackup', '--no-sync', + '--checkpoint' => 'fast', + '--pgdata' => $backupdir, + '--wal-method' => 'stream', + '--dbname' => $node->connstr('postgres'), + ], + '>' => \$stdout, + '2>' => \$stderr); + +# Wait until the walsender is about to create pg_basebackup's slot. +$node->wait_for_event('walsender', 'replication-slot-create-begin'); + +# Move WAL forward by several segments and checkpoint, so that every segment +# not protected by a slot or wal_keep_size is removed or recycled. +$node->advance_wal(3); +$node->safe_psql('postgres', 'CHECKPOINT'); + +$node->safe_psql('postgres', + "SELECT injection_points_detach('replication-slot-create-begin')"); +$node->safe_psql('postgres', + "SELECT injection_points_wakeup('replication-slot-create-begin')"); + +my $ok = $h->finish; +ok($ok, 'pg_basebackup succeeds with WAL activity during slot creation') + or diag("pg_basebackup stderr:\n$stderr"); +unlike( + $stderr, + qr/has already been removed/, + 'no WAL needed by the backup was removed'); + +# pg_verifybackup also parses the WAL the backup needs, so this fails if +# any of it is missing. +$node->command_ok([ 'pg_verifybackup', $backupdir ], + 'backup with streamed WAL verifies'); + +done_testing(); -- 2.56.0
^ permalink raw reply [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-09 13:19  Jesper St John <jesper@stjohn.se>
  parent: Kirill Reshke <reshkekirill@gmail.com>
  1 sibling, 2 replies; 11+ messages in thread

From: Jesper St John @ 2026-11-09 13:19 UTC (permalink / raw)
  To: Kirill Reshke <reshkekirill@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org

Added this patch to the CF app.

Made it into one and rebased it.

//Jesper

On 2026-10-22 07:44, Kirill Reshke wrote:
> On Wed, 21 Oct 2026 at 22:16, Jesper St John <jesper@stjohn.se> wrote:
>> 
>> On 2026-10-20 23:11, Kirill Reshke wrote:
> 
>> >
>> >> I've included a patch that moves the replication slot handing to run
>> >> before the backup part as a separate function.
>> >>
>> >> Some possible caveats already:
>> >> * Since moving the slot creation, we might run in to trouble if for
>> >> example, --checkpoint=spread is used since the slot is open for
>> >> longer.
>> >> * If any error arises in the backup part, now there will be a
>> >> lingering
>> >> slot when previously no slot was created.
>> >
>> > lingering slots looks like a real regression here. can we remove them
>> > when basebackup client disconnects?
>> 
>> A lingering slot could also happen today - only with this patch 
>> there's
>> a higher risk of it happening.
> 
> Higher risk of lingering slots is also a regression. But looks like
> this risk is anyway close to zero.
> 
>> Would some cleanup code in a function and use it with atexit() work?
> 
> Something like this should do, can be made a separate patch thought
> 
>> >
>> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
>> >
>> > + /* Get a second connection */
>> > + bgconn = GetConnection();
>> >
>> 
>> I've moved the code to its own function - the comment were there
>> already.
>> 
> 
> Yep, the point is It looks very out-of-context here. Like, the first
> line of the function says we're taking a second connection. Second to
> what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
> connection' or something.

Attachments:

v3-0001-pg_basebackup-Create-the-WAL-streaming-slot-befor.patch (text/x-diff, 13.1K) download From fc95dd067f316913a8e0f75dca63c4dcf3bde0a0 Mon Sep 17 00:00:00 2001 From: Jesper St John <jesper@stjohn.se> Date: Mon, 19 Oct 2026 15:35:59 +0200 Subject: [PATCH v3] pg_basebackup: Create the WAL streaming slot before starting the backup With --wal-method=stream, pg_basebackup created its replication slot only after BASE_BACKUP had returned the backup's start position. A checkpoint running in between could remove WAL from that position onwards, and the backup then failed with "requested WAL segment ... has already been removed". Open the WAL streaming connection and create its slot before sending BASE_BACKUP, rather than after it has returned the start position, so the slot reserves all the WAL the backup needs. Add a new TAP test that uses an injection point to hold slot creation while WAL is advanced and a checkpoint is run. Register an atexit handler, once the slot has been created, that drops it if pg_basebackup exits before the background WAL receiver has taken over the connection. Add tests to existing 010_pg_basebackup.pl that use a bogus --manifest-checksums value to fail BASE_BACKUP after the slot has been created, and check that only the new slot is dropped. --- src/bin/pg_basebackup/Makefile | 4 +- src/bin/pg_basebackup/meson.build | 2 + src/bin/pg_basebackup/pg_basebackup.c | 119 +++++++++++++----- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 42 +++++++ .../pg_basebackup/t/013_wal_slot_timing.pl | 82 ++++++++++++ 5 files changed, 214 insertions(+), 35 deletions(-) create mode 100644 src/bin/pg_basebackup/t/013_wal_slot_timing.pl diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile index df94fc27d02..12ff32e329b 100644 --- a/src/bin/pg_basebackup/Makefile +++ b/src/bin/pg_basebackup/Makefile @@ -12,13 +12,15 @@ PGFILEDESC = "pg_basebackup/pg_receivewal/pg_recvlogical - streaming WAL and backup receivers" PGAPPICON=win32 -EXTRA_INSTALL=contrib/test_decoding +EXTRA_INSTALL=contrib/test_decoding \ + src/test/modules/injection_points subdir = src/bin/pg_basebackup top_builddir = ../../.. include $(top_builddir)/src/Makefile.global # make these available to TAP test scripts +export enable_injection_points export LZ4 export TAR # Note that GZIP cannot be used directly as this environment variable is diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build index 2d3b22ff06e..43905ed9fc3 100644 --- a/src/bin/pg_basebackup/meson.build +++ b/src/bin/pg_basebackup/meson.build @@ -96,11 +96,13 @@ tests += { 'env': {'GZIP_PROGRAM': gzip.found() ? gzip.full_path() : '', 'TAR': tar.found() ? tar.full_path() : '', 'LZ4': program_lz4.found() ? program_lz4.full_path() : '', + 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no', }, 'tests': [ 't/010_pg_basebackup.pl', 't/011_in_place_tablespace.pl', 't/012_incremental_progress.pl', + 't/013_wal_slot_timing.pl', 't/020_pg_receivewal.pl', 't/030_pg_recvlogical.pl', 't/040_pg_createsubscriber.pl', diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c index c3b87a19e76..86144f01aa4 100644 --- a/src/bin/pg_basebackup/pg_basebackup.c +++ b/src/bin/pg_basebackup/pg_basebackup.c @@ -165,6 +165,9 @@ static bool found_existing_xlogdir = false; static bool made_tablespace_dirs = false; static bool found_tablespace_dirs = false; +/* Connection for the background WAL receiver, see PrepareLogStreamer() */ +static PGconn *bgconn = NULL; + /* Progress indicators */ static uint64 totalsize_kb; static uint64 totaldone; @@ -314,6 +317,22 @@ kill_bgchild_atexit(void) } #endif +/* + * Drop the lingering slot created by PrepareLogStreamer() if we fail before the + * background WAL receiver has taken over its connection. Only registered + * once the slot exists, so a pre-existing slot of the same name should never + * be dropped. + */ +static void +drop_lingering_replication_slot_atexit(void) +{ + /* Check that we have a good state */ + if (success || in_log_streamer || bgchild > 0) + return; + + (void) DropReplicationSlot(bgconn, replication_slot); +} + /* * Split argument into old_dir and new_dir and append to tablespace mapping * list. @@ -607,6 +626,62 @@ LogStreamerMain(logstreamer_param *param) return 0; } +/* + * Open the connection for the background WAL receiver and create its + * replication slot, if one is needed. + * + * This must be done before BASE_BACKUP is sent: the slot reserves WAL from + * its creation onwards, so creating it first guarantees that no WAL from the + * backup's start position onwards can be removed before the receiver starts + * streaming. If the slot were created only after BASE_BACKUP had returned + * the start position, a checkpoint in between could remove WAL the backup + * needs. + */ +static void +PrepareLogStreamer(void) +{ + /* Get a WAL serving connection */ + bgconn = GetConnection(); + if (!bgconn) + /* Error message already written in GetConnection() */ + exit(1); + + /* Temporary replication slots are only supported in 10 and newer */ + if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) + temp_replication_slot = false; + + /* + * Create replication slot if requested + */ + if (temp_replication_slot && !replication_slot) + replication_slot = psprintf("pg_basebackup_%u", + (unsigned int) PQbackendPID(bgconn)); + if (temp_replication_slot || create_slot) + { + if (!CreateReplicationSlot(bgconn, replication_slot, NULL, + temp_replication_slot, true, true, false, + false, false)) + exit(1); + + /* + * Only register atexit function if create_slot is true. Temporary + * slots go away with connection + */ + if (create_slot) + atexit(drop_lingering_replication_slot_atexit); + + if (verbose) + { + if (temp_replication_slot) + pg_log_info("created temporary replication slot \"%s\"", + replication_slot); + else + pg_log_info("created replication slot \"%s\"", + replication_slot); + } + } +} + /* * Initiate background process for receiving xlog during the backup. * The background stream will use its own database connection so we can @@ -620,7 +695,10 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, logstreamer_param *param; char statusdir[MAXPGPATH]; + Assert(bgconn != NULL); + param = pg_malloc0_object(logstreamer_param); + param->bgconn = bgconn; param->timeline = timeline; param->sysidentifier = sysidentifier; param->wal_compress_algorithm = wal_compress_algorithm; @@ -639,46 +717,12 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier, pg_fatal("could not create pipe for background process: %m"); #endif - /* Get a second connection */ - param->bgconn = GetConnection(); - if (!param->bgconn) - /* Error message already written in GetConnection() */ - exit(1); - /* In post-10 cluster, pg_xlog has been renamed to pg_wal */ snprintf(param->xlog, sizeof(param->xlog), "%s/%s", basedir, PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ? "pg_xlog" : "pg_wal"); - /* Temporary replication slots are only supported in 10 and newer */ - if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_TEMP_SLOTS) - temp_replication_slot = false; - - /* - * Create replication slot if requested - */ - if (temp_replication_slot && !replication_slot) - replication_slot = psprintf("pg_basebackup_%u", - (unsigned int) PQbackendPID(param->bgconn)); - if (temp_replication_slot || create_slot) - { - if (!CreateReplicationSlot(param->bgconn, replication_slot, NULL, - temp_replication_slot, true, true, false, - false, false)) - exit(1); - - if (verbose) - { - if (temp_replication_slot) - pg_log_info("created temporary replication slot \"%s\"", - replication_slot); - else - pg_log_info("created replication slot \"%s\"", - replication_slot); - } - } - if (format == 'p') { /* @@ -1970,6 +2014,13 @@ BaseBackup(char *compression_algorithm, char *compression_detail, compression_detail); } + /* + * If streaming WAL, create the replication slot before starting the + * backup, so that it protects all the WAL the backup needs. + */ + if (includewal == STREAM_WAL) + PrepareLogStreamer(); + if (verbose) pg_log_info("initiating base backup, waiting for checkpoint to complete"); diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index 23a757b1c86..a5c05aec1bf 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -829,9 +829,51 @@ $node->command_fails_like( qr/replication slot "slot0" already exists/, 'pg_basebackup fails with --create-slot --slot and a previously existing slot' ); +is( $node->safe_psql( + 'postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'slot0'} + ), + 'slot0', + 'previously existing slot is not dropped when --create-slot fails'); + +# The slot is created before BASE_BACKUP is sent, so a failure of +# BASE_BACKUP itself must not leave a slot created by --create-slot behind. +$node->command_fails_like( + [ + @pg_basebackup_defs, + '--pgdata' => "$tempdir/backupxs_slot2", + '--create-slot', + '--slot' => 'slot2', + '--manifest-checksums' => 'bogus' + ], + qr/unrecognized checksum algorithm/, + 'pg_basebackup --create-slot fails when BASE_BACKUP fails'); +is( $node->safe_psql( + 'postgres', + q{SELECT count(*) FROM pg_replication_slots WHERE slot_name = 'slot2'} + ), + '0', + 'slot created by failed pg_basebackup --create-slot is dropped'); $node->safe_psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')}); + +# Without --create-slot, a failed backup must leave the given slot alone. +$node->command_fails_like( + [ + @pg_basebackup_defs, + '--pgdata' => "$tempdir/backupxs_slot1_fail", + '--slot' => 'slot1', + '--manifest-checksums' => 'bogus' + ], + qr/unrecognized checksum algorithm/, + 'pg_basebackup --slot fails when BASE_BACKUP fails'); +is( $node->safe_psql( + 'postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'slot1'} + ), + 'slot1', + 'existing slot is not dropped when pg_basebackup --slot fails'); my $lsn = $node->safe_psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'} ); diff --git a/src/bin/pg_basebackup/t/013_wal_slot_timing.pl b/src/bin/pg_basebackup/t/013_wal_slot_timing.pl new file mode 100644 index 00000000000..0de5e24058b --- /dev/null +++ b/src/bin/pg_basebackup/t/013_wal_slot_timing.pl @@ -0,0 +1,82 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Check that pg_basebackup's replication slot reserves WAL from before the +# backup's start position. If the slot were created only after BASE_BACKUP +# had returned the start position, a checkpoint running in between could +# remove the WAL the backup needs, and streaming it would fail. +# +# We hold the slot creation with an injection point, force WAL past several +# segments and checkpoint, then let the slot creation continue. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +if ($ENV{enable_injection_points} ne 'yes') +{ + plan skip_all => 'Injection points not supported by this build'; +} + +my $tempdir = PostgreSQL::Test::Utils::tempdir; + +my $node = PostgreSQL::Test::Cluster->new('main'); +$node->init(allows_streaming => 1); +$node->append_conf( + 'postgresql.conf', q{ +wal_keep_size = 0 +checkpoint_timeout = 1h +}); +$node->start; + +if (!$node->check_extension('injection_points')) +{ + plan skip_all => 'Extension injection_points not installed'; +} + +$node->safe_psql('postgres', 'CREATE EXTENSION injection_points'); +$node->safe_psql('postgres', + "SELECT injection_points_attach('replication-slot-create-begin', 'wait')" +); + +my $backupdir = "$tempdir/backup"; +my ($stdout, $stderr); +my $h = IPC::Run::start( + [ + 'pg_basebackup', '--no-sync', + '--checkpoint' => 'fast', + '--pgdata' => $backupdir, + '--wal-method' => 'stream', + '--dbname' => $node->connstr('postgres'), + ], + '>' => \$stdout, + '2>' => \$stderr); + +# Wait until the walsender is about to create pg_basebackup's slot. +$node->wait_for_event('walsender', 'replication-slot-create-begin'); + +# Move WAL forward by several segments and checkpoint, so that every segment +# not protected by a slot or wal_keep_size is removed or recycled. +$node->advance_wal(3); +$node->safe_psql('postgres', 'CHECKPOINT'); + +$node->safe_psql('postgres', + "SELECT injection_points_detach('replication-slot-create-begin')"); +$node->safe_psql('postgres', + "SELECT injection_points_wakeup('replication-slot-create-begin')"); + +my $ok = $h->finish; +ok($ok, 'pg_basebackup succeeds with WAL activity during slot creation') + or diag("pg_basebackup stderr:\n$stderr"); +unlike( + $stderr, + qr/has already been removed/, + 'no WAL needed by the backup was removed'); + +# pg_verifybackup also parses the WAL the backup needs, so this fails if +# any of it is missing. +$node->command_ok([ 'pg_verifybackup', $backupdir ], + 'backup with streamed WAL verifies'); + +done_testing(); -- 2.56.0
^ permalink raw reply [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-12 16:12  Joao Rito <joaovictor.rito@gmail.com>
  parent: Jesper St John <jesper@stjohn.se>
  1 sibling, 0 replies; 11+ messages in thread

From: Joao Rito @ 2026-11-12 16:12 UTC (permalink / raw)
  To: Jesper St John <jesper@stjohn.se>; +Cc: pgsql-bugs@lists.postgresql.org, Kirill Reshke <reshkekirill@gmail.com>

Hi Jesper,

Thank you for the patch. I tested v3-0001 (CF entry 7613) and here are my
results.

Setup: master at c1694b7 (11 Nov 2026), configured with --enable-cassert,
--enable-tap-tests and --enable-injection-points, on Linux (WSL Ubuntu). The
patch applied cleanly with git apply.

1. The bug on unpatched master

I held the walsender at the existing injection point
replication-slot-create-begin (so BASE_BACKUP has returned, but the slot
does not exist yet). Then I switched WAL four times, ran CHECKPOINT, and
released the walsender. The server was running with wal_keep_size=0.

  pg_basebackup: error: unexpected termination of replication stream: ERROR:
  requested WAL segment 000000010000000000000002 has already been removed
  pg_basebackup: error: background process terminated unexpectedly

pg_basebackup exited with 1. So the race you describe is real, and it can be
triggered on demand.

2. With v3 applied

The same script gives pg_basebackup exit 0, no error in stderr or in the
server log, "backup successfully verified" from pg_verifybackup, and no
leftover slot.

3. Tests

"make -C src/bin/pg_basebackup check" with the patch: all 7 files pass (372
tests), including the new 013_wal_slot_timing.pl and the new cases in
010_pg_basebackup.pl. I ran 013 alone with prove -v to confirm it was not
skipped: "ok 1 - pg_basebackup succeeds with WAL activity during slot
creation", "ok 2 - no WAL needed by the backup was removed", "ok 3 - backup
with streamed WAL verifies", Files=1, Tests=3, Result: PASS. As a negative
control, I kept the new tests but put back the original pg_basebackup.c.
Then 013 fails all 3 subtests with the same "has already been removed"
error, so the test does detect the bug.

4. A note on lingering slots and signals

You and Kirill discussed the lingering slot in the thread. I tried to see
what happens on a signal. With v3 the slot is created before the checkpoint,
so the window in which an interrupted pg_basebackup can leave the slot
behind now also covers the checkpoint wait. With --checkpoint=spread that
can be long.

To compare fairly, I did the test after the slot exists on both builds (on
master the slot is only created after BASE_BACKUP returns, so a SIGTERM
during the checkpoint finds no slot there). I ran pg_basebackup -Xstream
--create-slot -S s1 --checkpoint=fast --max-rate=1M on a 216 MB database,
polled pg_replication_slots until s1 appeared, and then sent SIGTERM while
the backup was still running:

- master: s1 was still there after SIGTERM (and still there 11 s later).
- with v3: s1 was still there after SIGTERM (and still there 11 s later).

So master also leaves the slot behind on SIGTERM once the slot exists. I
think this is expected, since an atexit() handler probably does not run when
the process is killed by a signal, but I did not test that part directly. v3
does not make this worse for a signal after the slot is created. It only
starts the window earlier, so it is longer in time (the checkpoint wait).
The slot is not temporary here because I used --create-slot. The walsender
state differed between runs: in the first run s1 showed as inactive right
after the kill with v3 (active on master), and in a second run s1 was still
active on both builds 11 s after the kill.

Your atexit() handler works for the error cases covered by the TAP tests. If
you want to cover signals too, maybe a signal handler could drop the slot,
or the docs for --create-slot / --checkpoint could say that a slot may
remain if the process is killed. This is just a suggestion, I do not see it
as a regression of the patch. I only tested SIGTERM.

Small comment on the code: the check "success || in_log_streamer || bgchild
> 0" looks fine to me in the cases I ran, but I did not review all exit
paths in detail.

What I did NOT test:
- Other platforms (Windows, where pg_basebackup uses a thread instead of
  fork, and the new atexit() and bgconn handling matter).
- Other WAL methods and formats (--wal-method=fetch/none, tar format,
  compressed WAL, --format=tar with stream) beyond what the existing TAP
  suite covers.
- Older server versions (the MINIMUM_VERSION_FOR_TEMP_SLOTS branch).
- SIGINT, and failures after the background WAL receiver has started.
- Meson build (I used make/autoconf only), and the docs/pgindent formatting.
- The real two-node ansible scenario from your first mail (I only reproduced
  the race with an injection point), and the performance effect of holding
  the slot during long spread checkpoints (only the slot visibility was
  checked, above).
- Whether other tools that use the same pattern (for example pg_receivewal
  --create-slot) need the same change.

The scripts and the full log are available if you want them. Thanks again
for working on this.

Best regards,
Joao Rito






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-12 21:19  Zsolt Parragi <zsolt.parragi@percona.com>
  parent: Jesper St John <jesper@stjohn.se>
  1 sibling, 1 reply; 11+ messages in thread

From: Zsolt Parragi @ 2026-11-12 21:19 UTC (permalink / raw)
  To: Jesper St John <jesper@stjohn.se>; +Cc: pgsql-bugs@lists.postgresql.org, Kirill Reshke <reshkekirill@gmail.com>

Hello

I can also confirm the race on master and that v3 fixes it. However:

> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.

Moving the slot creation in front of BASE_BACKUP causes other new
failures, exactly like the concern you mentioned in the first email,
and I don't think overall it is better than the current issue.

1. The new slot can be invalidated by the backup's own checkpoint

A physical slot starts with the redo pointer of the previous
checkpoint. With the patch it is created, released (so it is
inactive), and stays like that for the whole start-of-backup
checkpoint. At the end of that checkpoint we call
InvalidateObsoleteReplicationSlots(), and our slot is one of the
candidates for invalidation, and can be invalidated based on
max_slot_wal_keep_size or idle_replication_slot_timeout.

This results in backup failures in multiple scenarios, with server log
entries such as

DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes.

2. The second connection is killed by idle_session_timeout

bgconn is now opened before BASE_BACKUP and sits idle in walsender
command mode until the checkpoint is done. PostgresMain() arms
IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
idle_session_timeout set and a checkpoint longer than that, the
walsender exits (dropping the temporary slot, and with it the
protection the patch adds), and the backup fails after the checkpoint:

pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
FATAL: terminating connection due to idle-session timeout

(2) is fixable, but (1) is a more tricky problem and I trickier to
solve. Maybe immediately after the slot is created we could use
READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?

Alternatively, instead of reordering, since this is an unlikely race,
maybe early detection and retry could be also an option instead of
reordering the operations? After creating the slot, check restart_lsn:
if it is behond startptr's segment then drop the connection and
restart.






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-16 10:26  Magnus Hagander <magnus@hagander.net>
  parent: Zsolt Parragi <zsolt.parragi@percona.com>
  0 siblings, 1 reply; 11+ messages in thread

From: Magnus Hagander @ 2026-11-16 10:26 UTC (permalink / raw)
  To: Zsolt Parragi <zsolt.parragi@percona.com>; +Cc: Jesper St John <jesper@stjohn.se>; pgsql-bugs@lists.postgresql.org, Kirill Reshke <reshkekirill@gmail.com>

On Thu, 12 Nov 2026 at 22:19, Zsolt Parragi <zsolt.parragi@percona.com>
wrote:

> Hello
>
> I can also confirm the race on master and that v3 fixes it. However:
>
> * Since moving the slot creation, we might run in to trouble if for
> > example, --checkpoint=spread is used since the slot is open for longer.
>
> Moving the slot creation in front of BASE_BACKUP causes other new
> failures, exactly like the concern you mentioned in the first email,
> and I don't think overall it is better than the current issue.
>
> 1. The new slot can be invalidated by the backup's own checkpoint
>
> A physical slot starts with the redo pointer of the previous
> checkpoint. With the patch it is created, released (so it is
> inactive), and stays like that for the whole start-of-backup
> checkpoint. At the end of that checkpoint we call
> InvalidateObsoleteReplicationSlots(), and our slot is one of the
> candidates for invalidation, and can be invalidated based on
> max_slot_wal_keep_size or idle_replication_slot_timeout.
>
> This results in backup failures in multiple scenarios, with server log
> entries such as
>
> DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672
> bytes.


> 2. The second connection is killed by idle_session_timeout
>
> bgconn is now opened before BASE_BACKUP and sits idle in walsender
> command mode until the checkpoint is done. PostgresMain() arms
> IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
> idle_session_timeout set and a checkpoint longer than that, the
> walsender exits (dropping the temporary slot, and with it the
> protection the patch adds), and the backup fails after the checkpoint:
>
> pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
> FATAL: terminating connection due to idle-session timeout
>
> (2) is fixable, but (1) is a more tricky problem and I trickier to
> solve. Maybe immediately after the slot is created we could use
> READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?
>
> Alternatively, instead of reordering, since this is an unlikely race,
> maybe early detection and retry could be also an option instead of
> reordering the operations? After creating the slot, check restart_lsn:
> if it is behond startptr's segment then drop the connection and
> restart.
>

Per some off-list discussion with Jesper. I don't think that wil work, as
READ_REPLICATION_SLOT won't actually advance the last use time on it, so it
would still time out.

I think the correct thing to do here is to start the full log streamer
*before* starting the base backup. This will lead to including just a
little bit unnecessarily much WAL in the backup, but it won't be much - we
already need to keep the WAL back to the start of the base backup
checkpoint start. So the only extra WAL to be included would be between the
log streaming started and the base backup command running, which would not
be much. The only real difference would be the streaming starts already
during the checkpoint, so with a slow cheakpoing it won't start the
streamer with a large backlog.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ <http://www.hagander.net/;
 Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/;

^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-16 12:08  Andrey Borodin <x4mmm@yandex-team.ru>
  parent: Magnus Hagander <magnus@hagander.net>
  0 siblings, 1 reply; 11+ messages in thread

From: Andrey Borodin @ 2026-11-16 12:08 UTC (permalink / raw)
  To: Magnus Hagander <magnus@hagander.net>; +Cc: Zsolt Parragi <zsolt.parragi@percona.com>; Jesper St John <jesper@stjohn.se>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Kirill Reshke <reshkekirill@gmail.com>

Hi Magnus, all,

Nick reported the same startpoint/slot race in August [0], tracked in
CF 7233 [1]. The client patch there opens the streaming connection and
creates its slot before BASE_BACKUP, but starts streaming only after
receiving the backup startpoint. Your proposal goes further than that.

We also discussed cleanup of a newly created permanent slot when
BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and
idle timeout are relevant to that approach too.

Could we combine the work and keep both threads linked from one CF
entry? This thread is currently tracked separately as CF 7613 [2].

Thank you!

Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail...
[1] https://commitfest.postgresql.org/patch/7233/
[2] https://commitfest.postgresql.org/patch/7613/






^ permalink  raw  reply  [nested|flat] 11+ messages in thread

* Re: Timing issue in pg_basebackup
@ 2026-11-17 11:35  Jesper St John <admin@zonezero.se>
  parent: Andrey Borodin <x4mmm@yandex-team.ru>
  0 siblings, 0 replies; 11+ messages in thread

From: Jesper St John @ 2026-11-17 11:35 UTC (permalink / raw)
  To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Magnus Hagander <magnus@hagander.net>; Zsolt Parragi <zsolt.parragi@percona.com>; Jesper St John <jesper@stjohn.se>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Kirill Reshke <reshkekirill@gmail.com>

Hi!

Sorry, even though I’m pretty sure I searched for similar issues before posting, I missed it.

Big thanks to Zsolt, Joao and Kirill for your input.

I think combining work sounds like the reasonable thing to do. I’d be happy to help out in either thread.

If any of you already know if one patch is preferable over the other, please do tell. If not I’ll review and try posting a follow-up.

I’ve slowly started to implement a “start streaming directly after slot creation”-functionality and perhaps that could work with either patch.

//Jesper


Sent from my iPhone

> On 16 Nov 2026, at 13:09, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> Hi Magnus, all,
> 
> Nick reported the same startpoint/slot race in August [0], tracked in
> CF 7233 [1]. The client patch there opens the streaming connection and
> creates its slot before BASE_BACKUP, but starts streaming only after
> receiving the backup startpoint. Your proposal goes further than that.
> 
> We also discussed cleanup of a newly created permanent slot when
> BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and
> idle timeout are relevant to that approach too.
> 
> Could we combine the work and keep both threads linked from one CF
> entry? This thread is currently tracked separately as CF 7613 [2].
> 
> Thank you!
> 
> Best regards, Andrey Borodin.
> 
> [0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail...
> [1] https://commitfest.postgresql.org/patch/7233/
> [2] https://commitfest.postgresql.org/patch/7613/






^ permalink  raw  reply  [nested|flat] 11+ messages in thread


end of thread, other threads:[~2026-11-17 11:35 UTC | newest]

Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-19 14:33 Timing issue in pg_basebackup Jesper St John <jesper@stjohn.se>
2026-10-20 21:11 ` Kirill Reshke <reshkekirill@gmail.com>
2026-10-21 17:16   ` Jesper St John <jesper@stjohn.se>
2026-10-22 05:44     ` Kirill Reshke <reshkekirill@gmail.com>
2026-10-29 19:26       ` Jesper St John <jesper@stjohn.se>
2026-11-09 13:19       ` Jesper St John <jesper@stjohn.se>
2026-11-12 16:12         ` Joao Rito <joaovictor.rito@gmail.com>
2026-11-12 21:19         ` Zsolt Parragi <zsolt.parragi@percona.com>
2026-11-16 10:26           ` Magnus Hagander <magnus@hagander.net>
2026-11-16 12:08             ` Andrey Borodin <x4mmm@yandex-team.ru>
2026-11-17 11:35               ` Jesper St John <admin@zonezero.se>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox