From: Michael Paquier <michael@paquier.xyz>
To: Victor Yegorov <vyegorov@gmail.com>
Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>
Cc: hubert depesz lubaczewski <depesz@depesz.com>
Cc: pgsql-bugs mailing list <pgsql-bugs@postgresql.org>
Subject: Re: Loss of replication after simple misconfiguration
Date: Fri, 10 Apr 2020 14:43:15 +0900
Message-ID: <20200410054315.GV1606@paquier.xyz> (raw)
In-Reply-To: <20200410041434.GU1606@paquier.xyz>
References: <20200409140836.GA5990@depesz.com>
<878sj4skmj.fsf@news-spur.riddles.org.uk>
<CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>
<20200410041434.GU1606@paquier.xyz>
On Fri, Apr 10, 2020 at 01:14:34PM +0900, Michael Paquier wrote:
> I have been able to write a TAP test to reproduce this exact scenario,
> though it succeeds for me (it could be a good idea to add some
> coverage for that actually..). Perhaps I am missing a step though?
> For example, perhaps the original setting was track_commit_timestamp =
> on, then it got disabled once?
I have been looking at more scenarios involving switching back and
forth track_commit_timestamps on/off during a switchover with a
standby failing to replay a XLOG_CHANGE_PARAMETER record during
recovery, still I cannot see a failure in scenarios close to what is
discussed here (the test includes transactions replayed with and
without the switch). Attached is a more advanced test. Any thoughts?
--
Michael
Attachments:committs-switchover-test-v2.patch (text/x-diff, 3.4K) downloaddiff --git a/src/test/modules/commit_ts/t/005_switchover.pl b/src/test/modules/commit_ts/t/005_switchover.plnew file mode 100644index 0000000000..e6db21f656--- /dev/null+++ b/src/test/modules/commit_ts/t/005_switchover.pl@@ -0,0 +1,95 @@+# Test master/standby switchover scenario where the track_commit_timestamp+# GUC is originally off, then enabled on the new primary after a switchover.+use strict;+use warnings;++use TestLib;+use Test::More tests => 3;+use PostgresNode;++my $bkplabel = 'backup';+my $master = get_new_node('master');+$master->init(allows_streaming => 1);+$master->append_conf(+ 'postgresql.conf', qq{+ track_commit_timestamp = on+ max_wal_senders = 5+ max_worker_processes = 8+ });+$master->start;+$master->backup($bkplabel);++my $standby = get_new_node('standby');+$standby->init_from_backup($master, $bkplabel, has_streaming => 1);+$standby->start;++for my $i (1 .. 10)+{+ $master->safe_psql('postgres', "create table t$i()");+}+my $master_lsn =+ $master->safe_psql('postgres', 'select pg_current_wal_lsn()');+$standby->poll_query_until('postgres',+ qq{SELECT '$master_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})+ or die "standby never caught up";++$standby->safe_psql('postgres', 'checkpoint');+$standby->restart;++# Begin the switchover, and first stop cleanly the primary.+$master->stop;++# Promote the standby with larger max_worker_processes, and enable+# track_commit_timestamp while on it. Generate a XLOG_PARAMETER_CHANGE+# record with those values.+$standby->promote;+$standby->append_conf('postgresql.conf', 'track_commit_timestamp = off');+$standby->append_conf('postgresql.conf', 'max_worker_processes = 10');+$standby->restart;++# Enable streaming on the previous primary, and start it. The node will+# shut down once it finds the record XLOG_PARAMETER_CHANGE generated+# previously.+$master->enable_streaming($standby);+command_fails(+ [ 'pg_ctl', '-D', $master->data_dir, '-l', $master->logfile, 'start' ],+ 'start fails because of max_worker_processes lower on standby at recovery');++# Consume some transactions on the standby without the primary knowing+# about it yet, generating some commit timestamps.+for my $i (11 .. 20)+{+ $standby->safe_psql('postgres', "create table t$i()");+}++# Toggle the configuration back on, and do more transactions.+$standby->append_conf('postgresql.conf', 'track_commit_timestamp = on');+$standby->restart;+for my $i (21 .. 30)+{+ $standby->safe_psql('postgres', "create table t$i()");+}++# Reconfigure the new standby and start it so as recovery is able to+# go through with a setting of max_worker_processes much larger than+# the primary.+$master->append_conf('postgresql.conf', 'max_worker_processes = 50');+$master->start;+my $standby_lsn =+ $standby->safe_psql('postgres', 'select pg_current_wal_lsn()');+$master->poll_query_until('postgres',+ qq{SELECT '$standby_lsn'::pg_lsn <= pg_last_wal_replay_lsn()});++# And finally check that commit timestamps are generated on the+# new primary.+my $standby_ts = $standby->safe_psql('postgres',+ qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't30'}+);+isnt($standby_ts, '',+ "standby gives valid non-empty value after promotion");++$standby_ts = $standby->safe_psql('postgres',+ qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't11'}+);+is($standby_ts, '',+ "standby gives valid empty value after promotion");
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, vyegorov@gmail.com, andrew@tao11.riddles.org.uk, depesz@depesz.com
Subject: Re: Loss of replication after simple misconfiguration
In-Reply-To: <20200410054315.GV1606@paquier.xyz>
References: <20200409140836.GA5990@depesz.com> <878sj4skmj.fsf@news-spur.riddles.org.uk> <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com> <20200410041434.GU1606@paquier.xyz> <20200410054315.GV1606@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox