From: Ronan Dunklau <ronan.dunklau@aiven.io>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>
Subject: Re: pg_receivewal starting position
Date: Tue, 26 Oct 2021 11:01:46 +0200
Message-ID: <22424666.6Emhk5qWAg@aivenronan> (raw)
In-Reply-To: <1914572.usQuhbGJ8B@aivenronan>
References: <CALj2ACW6YA0SfqwPm1-FRRdeNj1e9isiSax3jZ8MmyQYYQQWNw@mail.gmail.com>
<YXdWvIO52CiT8kg9@paquier.xyz>
<1914572.usQuhbGJ8B@aivenronan>
Le mardi 26 octobre 2021, 08:27:47 CEST Ronan Dunklau a écrit :
> Yes, I will try to simplify the logic of the patch I sent last week. I'll
> keep you posted here soon.
I was able to simplify it quite a bit, by using only one standby for both test
scenarios.
This test case verify that after a timeline switch, if we resume from a
previous state we will archive:
- segments from the old timeline
- segments from the new timeline
- the timeline history file itself.
I chose to check against a full segment from the previous timeline, but it
would have been possible to check that the latest timeline segment was
partial. I chose not not, in the unlikely event we promote at an exact segment
boundary. I don't think it matters much, since partial wal files are already
covered by other tests.
--
Ronan Dunklau
Attachments:v13-0001-Add-a-test-for-pg_receivewal-following-timeline-.patch (text/x-patch, 4.9K) download
From 0daf03e7e69aa11accb18df86d686004002115ff Mon Sep 17 00:00:00 2001
From: Ronan Dunklau <ronan.dunklau@aiven.io>
Date: Tue, 26 Oct 2021 10:54:12 +0200
Subject: [PATCH v13] Add a test for pg_receivewal following timeline switch.
pg_receivewal is able to follow a timeline switch, but this was not
tested anywher. This test case verify that it works as expected both
when resuming from a replication slot and from the archive directory,
which have different methods of retrieving the timeline it left off.
---
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 87 +++++++++++++++++++-
1 file changed, 86 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.plindex 092c9b6f25..f285e9b3f3 100644--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 31;+use Test::More tests => 39;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -206,3 +206,88 @@ $primary->command_ok(
"WAL streamed from the slot's restart_lsn");
ok(-e "$slot_dir/$walfile_streamed",
"WAL from the slot's restart_lsn has been archived");
++# Test a timeline switch.+# This test is split in two, using the same standby: one test check the+# resume-from-folder case, the other the resume-from-slot one.++# Setup a standby for our tests+my $backup_name = "basebackup";+$primary->backup($backup_name);+my $standby = PostgreSQL::Test::Cluster->new("standby");+$standby->init_from_backup($primary, $backup_name, has_streaming => 1);+$standby->start;++# Cleanup the previous stream directories to reuse them+unlink glob "'${stream_dir}/*'";+unlink glob "'${slot_dir}/*'";++# Create two replication slots.+# First one is to make sure we keep the wal for the resume-from-folder case.+my $folder_slot = "folder_slot";+$standby->psql('',+ "CREATE_REPLICATION_SLOT $folder_slot PHYSICAL (RESERVE_WAL)",+ replication => 1);+# Second one is for testing the resume-from-slot case+my $archive_slot = "archive_slot";+$standby->psql('',+ "CREATE_REPLICATION_SLOT $archive_slot PHYSICAL (RESERVE_WAL)",+ replication => 1);+# Get a walfilename from before the promotion to make sure it is archived+# after promotion+my $walfile_before_promotion = $primary->safe_psql('postgres',+ "SELECT pg_walfile_name(pg_current_wal_insert_lsn())");+# Switch wal to make sure it is not a partial file but a complete segment.+$primary->psql('postgres',+ 'INSERT INTO test_table VALUES (generate_series(1,100));');+$primary->psql('postgres', 'SELECT pg_switch_wal();');++# Populate the stream_dir for the resume-from-folder case.+$nextlsn = $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');+chomp($nextlsn);+$standby->run_log(+ [ 'pg_receivewal', '-D', $stream_dir, '--verbose', '--endpos', $nextlsn,+ '--slot', $folder_slot, '--no-sync'],+ "Stream some wal before promoting");++# Everything is setup, promote the standby to trigger a timeline switch+$standby->psql(+ 'postgres',+ "SELECT pg_promote(wait_seconds => 300)");++# Force a wal switch to make sure at least one full WAL is archived on the new+# timeline, and fetch this walfilename.+my $walfile_after_promotion = $standby->safe_psql('postgres',+ "SELECT pg_walfile_name(pg_current_wal_insert_lsn())");+$standby->psql('postgres',+ 'INSERT INTO test_table VALUES (generate_series(1,100));');+$standby->psql('postgres', 'SELECT pg_switch_wal();');+$nextlsn =+ $standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');+chomp($nextlsn);++# Now, try to resume after the promotion, from the folder.+$standby->command_ok(+ [ 'pg_receivewal', '-D', $stream_dir, '--verbose', '--endpos', $nextlsn,+ '--slot', $folder_slot, '--no-sync'],+ "Stream some wal after promoting, resuming from the folder's position");++ok(-e "$stream_dir/$walfile_before_promotion",+ "WAL from the old timeline has been archived resuming from the folder");+ok(-e "$stream_dir/$walfile_after_promotion",+ "WAL from the new timeline has been archived resuming from the folder");+ok(-e "$stream_dir/00000002.history",+ "Timeline history file has been archived resuming from the folder");++# Same thing, resuming from the slot+$standby->command_ok(+ [ 'pg_receivewal', '-D', $slot_dir, '--verbose', '--endpos', $nextlsn,+ '--slot', $archive_slot, '--no-sync'],+ "Stream some wal after promoting, resuming from the slot's position");+my @walfiles = glob "$slot_dir/*";+ok(-e "$slot_dir/$walfile_before_promotion",+ "WAL from the old timeline has been archived resuming from the slot");+ok(-e "$slot_dir/$walfile_after_promotion",+ "WAL from the new timeline has been archived resuming from the slot");+ok(-e "$slot_dir/00000002.history",+ "Timeline history file has been archived resuming from the slot");--
2.33.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: ronan.dunklau@aiven.io, bharath.rupireddyforpostgres@gmail.com, michael@paquier.xyz, pgsql-hackers@lists.postgresql.org, horikyota.ntt@gmail.com, sawada.mshk@gmail.com
Subject: Re: pg_receivewal starting position
In-Reply-To: <22424666.6Emhk5qWAg@aivenronan>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox