From: Michael Paquier <michael@paquier.xyz>
To: Victor Yegorov <vyegorov@gmail.com>
Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>
Cc: hubert depesz lubaczewski <depesz@depesz.com>
Cc: pgsql-bugs mailing list <pgsql-bugs@postgresql.org>
Subject: Re: Loss of replication after simple misconfiguration
Date: Fri, 10 Apr 2020 13:14:34 +0900
Message-ID: <20200410041434.GU1606@paquier.xyz> (raw)
In-Reply-To: <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>
References: <20200409140836.GA5990@depesz.com>
<878sj4skmj.fsf@news-spur.riddles.org.uk>
<CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>
On Thu, Apr 09, 2020 at 07:48:17PM +0300, Victor Yegorov wrote:
> We've hit similar issue last week, but on 11.5 — we
> had track_commit_timestamp enabled on master after switchover,
> replica failed to start.
>
> It might be, that fix was here:
> https://git.postgresql.org/pg/commitdiff/180feb8c7e
> (For 9.5 branch it is: https://git.postgresql.org/pg/commitdiff/69a5686368)
>
> We're not in the position to test it again, though…
Hmm. We have a gap in tests here as we don't have any tests stressing
switchovers when it comes to track_commit_timestamps. Anyway, could
you confirm that I got the problem right? Here is the flow I am getting
from the information of upthread, roughly:
1) Primary/standby cluster, both using max_worker_processes = 8, and
track_commit_timestamp = off.
2) In order to begin the switchover, first stop cleanly the primary.
3) Update configuration of the standby as follows, promote it and
restart it:
track_commit_timestamp = on
max_worker_processes = 50
4) Enable streaming on the old primary to make it a standby, starting
it fails because of the unmatching setting for max_worker_processes.
5) Re-adjust max_worker_processes correctly on the new standby, start
it. Then this startup should fail at the lookup of pg_commit_ts/.
I have been able to write a TAP test to reproduce this exact scenario,
though it succeeds for me (it could be a good idea to add some
coverage for that actually..). Perhaps I am missing a step though?
For example, perhaps the original setting was track_commit_timestamp =
on, then it got disabled once?
--
Michael
Attachments:committs-switchover-test.patch (text/x-diff, 3.2K) downloaddiff --git a/src/test/modules/commit_ts/t/005_switchover.pl b/src/test/modules/commit_ts/t/005_switchover.plnew file mode 100644index 0000000000..f414f632c5--- /dev/null+++ b/src/test/modules/commit_ts/t/005_switchover.pl@@ -0,0 +1,85 @@+# Test master/standby switchover scenario where the track_commit_timestamp+# GUC is originally off, then enabled on the new primary after a switchover.+use strict;+use warnings;++use TestLib;+use Test::More tests => 2;+use PostgresNode;++my $bkplabel = 'backup';+my $master = get_new_node('master');+$master->init(allows_streaming => 1);+$master->append_conf(+ 'postgresql.conf', qq{+ track_commit_timestamp = off+ max_wal_senders = 5+ max_worker_processes = 8+ });+$master->start;+$master->backup($bkplabel);++my $standby = get_new_node('standby');+$standby->init_from_backup($master, $bkplabel, has_streaming => 1);+$standby->start;++for my $i (1 .. 10)+{+ $master->safe_psql('postgres', "create table t$i()");+}+my $master_lsn =+ $master->safe_psql('postgres', 'select pg_current_wal_lsn()');+$standby->poll_query_until('postgres',+ qq{SELECT '$master_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})+ or die "standby never caught up";++$standby->safe_psql('postgres', 'checkpoint');+$standby->restart;++# Begin the switchover, stop the primary. Note that attempting to+# start this cluster with a value of max_worker_processes lower than+# the original causes its startup to fail.+$master->append_conf('postgresql.conf', 'max_worker_processes = 4');+$master->stop;++# Promote the standby with larger max_worker_processes, and enable+# track_commit_timestamp while on it.+$standby->append_conf('postgresql.conf', 'track_commit_timestamp = on');+$standby->promote;+$standby->restart;++# Enable streaming on the previous primary, and attempt to start it.+# This fails as max_worker_processes is lower than on the root+# cluster.+$master->enable_streaming($standby);+command_fails(+ [ 'pg_ctl', '-D', $master->data_dir, '-l', $master->logfile, 'start' ],+ 'start fails because of max_worker_processes lower on standby');++# Consume some transactions on the standby without the primary knowing+# about it yet.+for my $i (1 .. 100)+{+ $standby->safe_psql('postgres', "select txid_current();");+}+$standby->safe_psql('postgres', "create table t11()");++# Add now a proper setting for max_worker_processes to allow startup+# to go through. This new standby will replay all information+# received from the transactions of the standby previously done.+$master->append_conf('postgresql.conf', 'max_worker_processes = 8');+$master->start;++# Check that old master is correctly streaming as a new standby.+my $standby_lsn =+ $standby->safe_psql('postgres', 'select pg_current_wal_lsn()');+$master->poll_query_until('postgres',+ qq{SELECT '$standby_lsn'::pg_lsn <= pg_last_wal_replay_lsn()});++# And finally check that commit timestamps are generated on the+# new primary.+my $standby_ts = $standby->safe_psql('postgres',+ qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't11'}+);+isnt($standby_ts, '',+ "standby gives valid value ($standby_ts) after promotion");
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, vyegorov@gmail.com, andrew@tao11.riddles.org.uk, depesz@depesz.com
Subject: Re: Loss of replication after simple misconfiguration
In-Reply-To: <20200410041434.GU1606@paquier.xyz>
References: <20200409140836.GA5990@depesz.com> <878sj4skmj.fsf@news-spur.riddles.org.uk> <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com> <20200410041434.GU1606@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox