From: Drouvot, Bertrand <bertranddrouvot.pg@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com>
Cc: vignesh C <vignesh21@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Add two missing tests in 035_standby_logical_decoding.pl
Date: Tue, 2 May 2023 13:22:26 +0200
Message-ID: <8a13f0d8-71f3-773b-7135-5974fd53724b@gmail.com> (raw)
In-Reply-To: <CAA4eK1LEQfC-ZxOkN4=cuAxB1N=2iLcEEb+Mb=XGCqGhwvMkaA@mail.gmail.com>
References: <2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com>
<CALDaNm2raNJrEF_ucDeTV_AygYutVb1bz2SrKO06YdVVG1gf2A@mail.gmail.com>
<9ab1827c-9a57-8c48-59b2-c5fb62c21be3@gmail.com>
<CAA4eK1KVoAjzT49BwKCih0A+1PE7hm=4PwQ-_VUr9YteFJRB5g@mail.gmail.com>
<CAA4eK1JeAZ9MUf3qPAOA5t9=e_vPJSEi6nZ3OygDYi-913eO+Q@mail.gmail.com>
<74f80bc1-6b39-b2a8-c716-98beefb75e0b@gmail.com>
<OSZPR01MB6310F24417BDD04731BD7978FD659@OSZPR01MB6310.jpnprd01.prod.outlook.com>
<17b6b056-4f53-c444-0c94-869547d1a5df@gmail.com>
<CAA4eK1JiYisk1aqn1uvjd+VCfsVtXLby7uMmkLQU_ggAx7xt6w@mail.gmail.com>
<054defef-de15-53e4-6c5a-d16ccc21d4ba@gmail.com>
<CAA4eK1LEQfC-ZxOkN4=cuAxB1N=2iLcEEb+Mb=XGCqGhwvMkaA@mail.gmail.com>
Hi,
On 5/2/23 8:28 AM, Amit Kapila wrote:
> On Fri, Apr 28, 2023 at 2:24 PM Drouvot, Bertrand
> <bertranddrouvot.pg@gmail.com> wrote:
>>
>> I can see V7 failing on "Cirrus CI / macOS - Ventura - Meson" only (other machines are not complaining).
>>
>> It does fail on "invalidated logical slots do not lead to retaining WAL", see https://cirrus-ci.com/task/4518083541336064
>>
>> I'm not sure why it is failing, any idea?
>>
>
> I think the reason for the failure is that on standby, the test is not
> able to remove the file corresponding to the invalid slot. You are
> using pg_switch_wal() to generate a switch record and I think you need
> one more WAL-generating statement after that to achieve your purpose
> which is that during checkpoint, the tes removes the WAL file
> corresponding to an invalid slot. Just doing checkpoint on primary may
> not serve the need as that doesn't lead to any new insertion of WAL on
> standby. Is your v6 failing in the same environment?
Thanks for the feedback!
No V6 was working fine.
> If not, then it
> is probably due to the reason that the test is doing insert after
> pg_switch_wal() in that version. Why did you change the order of
> insert in v7?
>
I thought doing the insert before the switch was ok and as my local test
was running fine I did not re-consider the ordering.
> BTW, you can confirm the failure by changing the DEBUG2 message in
> RemoveOldXlogFiles() to LOG. In the case, where the test fails, it may
> not remove the WAL file corresponding to an invalid slot whereas it
> will remove the WAL file when the test succeeds.
Yeah, I added more debug information and what I can see is that the WAL file
we want to see removed is "000000010000000000000003" while the standby emits:
"
2023-05-02 10:03:28.351 UTC [16971][checkpointer] LOG: attempting to remove WAL segments older than log file 000000000000000000000002
2023-05-02 10:03:28.351 UTC [16971][checkpointer] LOG: recycled write-ahead log file "000000010000000000000002"
"
As per your suggestion, changing the insert ordering (like in V8 attached) makes it now work on the failing environment too.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.comAttachments:v8-0001-Add-a-test-to-verify-that-invalidated-logical-slo.patch (text/plain, 2.5K) download
From defaeb000b09eab9ba7b5d08c032f81b5bd72a53 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 28 Apr 2023 07:27:20 +0000
Subject: [PATCH v8] Add a test to verify that invalidated logical slots do not
lead to retaining WAL.
---
.../t/035_standby_logical_decoding.pl | 39 ++++++++++++++++++-
1 file changed, 37 insertions(+), 2 deletions(-)
100.0% src/test/recovery/t/
diff --git a/src/test/recovery/t/035_standby_logical_decoding.pl b/src/test/recovery/t/035_standby_logical_decoding.plindex 66d264f230..a6b640d7fd 100644--- a/src/test/recovery/t/035_standby_logical_decoding.pl+++ b/src/test/recovery/t/035_standby_logical_decoding.pl@@ -500,9 +500,44 @@ $node_standby->restart;
check_slots_conflicting_status(1);
##################################################
-# Verify that invalidated logical slots do not lead to retaining WAL+# Verify that invalidated logical slots do not lead to retaining WAL.
##################################################
-# XXXXX TODO++# Before removing WAL file(s), ensure the cascading standby catch up+$node_standby->wait_for_replay_catchup($node_cascading_standby,+ $node_primary);++# Get the restart_lsn from an invalidated slot+my $restart_lsn = $node_standby->safe_psql('postgres',+ "SELECT restart_lsn from pg_replication_slots WHERE slot_name = 'vacuum_full_activeslot' and conflicting is true;"+);++chomp($restart_lsn);++# As pg_walfile_name() can not be executed on the standby,+# get the WAL file name associated to this lsn from the primary+my $walfile_name = $node_primary->safe_psql('postgres',+ "SELECT pg_walfile_name('$restart_lsn')");++chomp($walfile_name);++# Generate some activity and switch WAL file on the primary+$node_primary->safe_psql(+ 'postgres', "create table retain_test(a int);+ select pg_switch_wal();+ insert into retain_test values(1);+ checkpoint;");++# Wait for the standby to catch up+$node_primary->wait_for_catchup($node_standby);++# Request a checkpoint on the standby to trigger the WAL file(s) removal+$node_standby->safe_psql('postgres', 'checkpoint;');++# Verify that the wal file has not been retained on the standby+my $standby_walfile = $node_standby->data_dir . '/pg_wal/' . $walfile_name;+ok(!-f "$standby_walfile",+ "invalidated logical slots do not lead to retaining WAL");
##################################################
# Recovery conflict: Invalidate conflicting slots, including in-use slots
--
2.34.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bertranddrouvot.pg@gmail.com, amit.kapila16@gmail.com, shiy.fnst@fujitsu.com, vignesh21@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Add two missing tests in 035_standby_logical_decoding.pl
In-Reply-To: <8a13f0d8-71f3-773b-7135-5974fd53724b@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox