pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Michael Paquier <michael@paquier.xyz>
To: Victor Yegorov <vyegorov@gmail.com>
Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>
Cc: hubert depesz lubaczewski <depesz@depesz.com>
Cc: pgsql-bugs mailing list <pgsql-bugs@postgresql.org>
Subject: Re: Loss of replication after simple misconfiguration
Date: Fri, 10 Apr 2020 13:14:34 +0900
Message-ID: <20200410041434.GU1606@paquier.xyz> (raw)
In-Reply-To: <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>
References: <20200409140836.GA5990@depesz.com>
	<878sj4skmj.fsf@news-spur.riddles.org.uk>
	<CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>

On Thu, Apr 09, 2020 at 07:48:17PM +0300, Victor Yegorov wrote:
> We've hit similar issue last week, but on 11.5 — we
> had track_commit_timestamp enabled on master after switchover,
> replica failed to start.
> 
> It might be, that fix was here:
> https://git.postgresql.org/pg/commitdiff/180feb8c7e
> (For 9.5 branch it is: https://git.postgresql.org/pg/commitdiff/69a5686368)
> 
> We're not in the position to test it again, though…

Hmm.  We have a gap in tests here as we don't have any tests stressing
switchovers when it comes to track_commit_timestamps.  Anyway, could
you confirm that I got the problem right?  Here is the flow I am getting
from the information of upthread, roughly:
1) Primary/standby cluster, both using max_worker_processes = 8, and
track_commit_timestamp = off.
2) In order to begin the switchover, first stop cleanly the primary.
3) Update configuration of the standby as follows, promote it and
restart it:
track_commit_timestamp = on
max_worker_processes = 50
4) Enable streaming on the old primary to make it a standby, starting
it fails because of the unmatching setting for max_worker_processes.
5) Re-adjust max_worker_processes correctly on the new standby, start
it.  Then this startup should fail at the lookup of pg_commit_ts/.

I have been able to write a TAP test to reproduce this exact scenario,
though it succeeds for me (it could be a good idea to add some
coverage for that actually..).  Perhaps I am missing a step though?
For example, perhaps the original setting was track_commit_timestamp =
on, then it got disabled once?
--
Michael

Attachments:

committs-switchover-test.patch (text/x-diff, 3.2K) download diff --git a/src/test/modules/commit_ts/t/005_switchover.pl b/src/test/modules/commit_ts/t/005_switchover.pl new file mode 100644 index 0000000000..f414f632c5 --- /dev/null +++ b/src/test/modules/commit_ts/t/005_switchover.pl @@ -0,0 +1,85 @@ +# Test master/standby switchover scenario where the track_commit_timestamp +# GUC is originally off, then enabled on the new primary after a switchover. +use strict; +use warnings; + +use TestLib; +use Test::More tests => 2; +use PostgresNode; + +my $bkplabel = 'backup'; +my $master = get_new_node('master'); +$master->init(allows_streaming => 1); +$master->append_conf( + 'postgresql.conf', qq{ + track_commit_timestamp = off + max_wal_senders = 5 + max_worker_processes = 8 + }); +$master->start; +$master->backup($bkplabel); + +my $standby = get_new_node('standby'); +$standby->init_from_backup($master, $bkplabel, has_streaming => 1); +$standby->start; + +for my $i (1 .. 10) +{ + $master->safe_psql('postgres', "create table t$i()"); +} +my $master_lsn = + $master->safe_psql('postgres', 'select pg_current_wal_lsn()'); +$standby->poll_query_until('postgres', + qq{SELECT '$master_lsn'::pg_lsn <= pg_last_wal_replay_lsn()}) + or die "standby never caught up"; + +$standby->safe_psql('postgres', 'checkpoint'); +$standby->restart; + +# Begin the switchover, stop the primary. Note that attempting to +# start this cluster with a value of max_worker_processes lower than +# the original causes its startup to fail. +$master->append_conf('postgresql.conf', 'max_worker_processes = 4'); +$master->stop; + +# Promote the standby with larger max_worker_processes, and enable +# track_commit_timestamp while on it. +$standby->append_conf('postgresql.conf', 'track_commit_timestamp = on'); +$standby->promote; +$standby->restart; + +# Enable streaming on the previous primary, and attempt to start it. +# This fails as max_worker_processes is lower than on the root +# cluster. +$master->enable_streaming($standby); +command_fails( + [ 'pg_ctl', '-D', $master->data_dir, '-l', $master->logfile, 'start' ], + 'start fails because of max_worker_processes lower on standby'); + +# Consume some transactions on the standby without the primary knowing +# about it yet. +for my $i (1 .. 100) +{ + $standby->safe_psql('postgres', "select txid_current();"); +} +$standby->safe_psql('postgres', "create table t11()"); + +# Add now a proper setting for max_worker_processes to allow startup +# to go through. This new standby will replay all information +# received from the transactions of the standby previously done. +$master->append_conf('postgresql.conf', 'max_worker_processes = 8'); +$master->start; + +# Check that old master is correctly streaming as a new standby. +my $standby_lsn = + $standby->safe_psql('postgres', 'select pg_current_wal_lsn()'); +$master->poll_query_until('postgres', + qq{SELECT '$standby_lsn'::pg_lsn <= pg_last_wal_replay_lsn()}); + +# And finally check that commit timestamps are generated on the +# new primary. +my $standby_ts = $standby->safe_psql('postgres', + qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't11'} +); +isnt($standby_ts, '', + "standby gives valid value ($standby_ts) after promotion");
signature.asc (application/pgp-signature, 832B) download

view thread (8+ messages)  latest in thread

Message-ID: <20200410041434.GU1606@paquier.xyz>
Permalink:  ../20200410041434.GU1606@paquier.xyz/
Also on:    postgresql.org/message-id/20200410041434.GU1606@paquier.xyz

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: michael@paquier.xyz, vyegorov@gmail.com, andrew@tao11.riddles.org.uk, depesz@depesz.com
  Subject: Re: Loss of replication after simple misconfiguration
  In-Reply-To: <20200410041434.GU1606@paquier.xyz>
  References: <20200409140836.GA5990@depesz.com> <878sj4skmj.fsf@news-spur.riddles.org.uk> <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com> <20200410041434.GU1606@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox