pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Michael Paquier <michael@paquier.xyz>
To: Victor Yegorov <vyegorov@gmail.com>
Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>
Cc: hubert depesz lubaczewski <depesz@depesz.com>
Cc: pgsql-bugs mailing list <pgsql-bugs@postgresql.org>
Subject: Re: Loss of replication after simple misconfiguration
Date: Fri, 10 Apr 2020 14:43:15 +0900
Message-ID: <20200410054315.GV1606@paquier.xyz> (raw)
In-Reply-To: <20200410041434.GU1606@paquier.xyz>
References: <20200409140836.GA5990@depesz.com>
	<878sj4skmj.fsf@news-spur.riddles.org.uk>
	<CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com>
	<20200410041434.GU1606@paquier.xyz>

On Fri, Apr 10, 2020 at 01:14:34PM +0900, Michael Paquier wrote:
> I have been able to write a TAP test to reproduce this exact scenario,
> though it succeeds for me (it could be a good idea to add some
> coverage for that actually..).  Perhaps I am missing a step though?
> For example, perhaps the original setting was track_commit_timestamp =
> on, then it got disabled once?

I have been looking at more scenarios involving switching back and
forth track_commit_timestamps on/off during a switchover with a
standby failing to replay a XLOG_CHANGE_PARAMETER record during
recovery, still I cannot see a failure in scenarios close to what is
discussed here (the test includes transactions replayed with and
without the switch).  Attached is a more advanced test.  Any thoughts?
--
Michael

Attachments:

committs-switchover-test-v2.patch (text/x-diff, 3.4K) download diff --git a/src/test/modules/commit_ts/t/005_switchover.pl b/src/test/modules/commit_ts/t/005_switchover.pl new file mode 100644 index 0000000000..e6db21f656 --- /dev/null +++ b/src/test/modules/commit_ts/t/005_switchover.pl @@ -0,0 +1,95 @@ +# Test master/standby switchover scenario where the track_commit_timestamp +# GUC is originally off, then enabled on the new primary after a switchover. +use strict; +use warnings; + +use TestLib; +use Test::More tests => 3; +use PostgresNode; + +my $bkplabel = 'backup'; +my $master = get_new_node('master'); +$master->init(allows_streaming => 1); +$master->append_conf( + 'postgresql.conf', qq{ + track_commit_timestamp = on + max_wal_senders = 5 + max_worker_processes = 8 + }); +$master->start; +$master->backup($bkplabel); + +my $standby = get_new_node('standby'); +$standby->init_from_backup($master, $bkplabel, has_streaming => 1); +$standby->start; + +for my $i (1 .. 10) +{ + $master->safe_psql('postgres', "create table t$i()"); +} +my $master_lsn = + $master->safe_psql('postgres', 'select pg_current_wal_lsn()'); +$standby->poll_query_until('postgres', + qq{SELECT '$master_lsn'::pg_lsn <= pg_last_wal_replay_lsn()}) + or die "standby never caught up"; + +$standby->safe_psql('postgres', 'checkpoint'); +$standby->restart; + +# Begin the switchover, and first stop cleanly the primary. +$master->stop; + +# Promote the standby with larger max_worker_processes, and enable +# track_commit_timestamp while on it. Generate a XLOG_PARAMETER_CHANGE +# record with those values. +$standby->promote; +$standby->append_conf('postgresql.conf', 'track_commit_timestamp = off'); +$standby->append_conf('postgresql.conf', 'max_worker_processes = 10'); +$standby->restart; + +# Enable streaming on the previous primary, and start it. The node will +# shut down once it finds the record XLOG_PARAMETER_CHANGE generated +# previously. +$master->enable_streaming($standby); +command_fails( + [ 'pg_ctl', '-D', $master->data_dir, '-l', $master->logfile, 'start' ], + 'start fails because of max_worker_processes lower on standby at recovery'); + +# Consume some transactions on the standby without the primary knowing +# about it yet, generating some commit timestamps. +for my $i (11 .. 20) +{ + $standby->safe_psql('postgres', "create table t$i()"); +} + +# Toggle the configuration back on, and do more transactions. +$standby->append_conf('postgresql.conf', 'track_commit_timestamp = on'); +$standby->restart; +for my $i (21 .. 30) +{ + $standby->safe_psql('postgres', "create table t$i()"); +} + +# Reconfigure the new standby and start it so as recovery is able to +# go through with a setting of max_worker_processes much larger than +# the primary. +$master->append_conf('postgresql.conf', 'max_worker_processes = 50'); +$master->start; +my $standby_lsn = + $standby->safe_psql('postgres', 'select pg_current_wal_lsn()'); +$master->poll_query_until('postgres', + qq{SELECT '$standby_lsn'::pg_lsn <= pg_last_wal_replay_lsn()}); + +# And finally check that commit timestamps are generated on the +# new primary. +my $standby_ts = $standby->safe_psql('postgres', + qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't30'} +); +isnt($standby_ts, '', + "standby gives valid non-empty value after promotion"); + +$standby_ts = $standby->safe_psql('postgres', + qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't11'} +); +is($standby_ts, '', + "standby gives valid empty value after promotion");
signature.asc (application/pgp-signature, 832B) download

view thread (8+ messages)  latest in thread

Message-ID: <20200410054315.GV1606@paquier.xyz>
Permalink:  ../20200410054315.GV1606@paquier.xyz/
Also on:    postgresql.org/message-id/20200410054315.GV1606@paquier.xyz

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: michael@paquier.xyz, vyegorov@gmail.com, andrew@tao11.riddles.org.uk, depesz@depesz.com
  Subject: Re: Loss of replication after simple misconfiguration
  In-Reply-To: <20200410054315.GV1606@paquier.xyz>
  References: <20200409140836.GA5990@depesz.com> <878sj4skmj.fsf@news-spur.riddles.org.uk> <CAGnEboj9igBHb8hF2Ls4oNOJJ2EN3xBRm__SXg1i_N+T+v=WEQ@mail.gmail.com> <20200410041434.GU1606@paquier.xyz> <20200410054315.GV1606@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox