pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Michael Paquier <michael@paquier.xyz>
To: Alexander Kukushkin <cyberdemn@gmail.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15346: Replica fails to start after the crash
Date: Wed, 29 Aug 2018 05:10:51 -0700
Message-ID: <20180829121051.GC5903@paquier.xyz> (raw)
In-Reply-To: <CAFh8B=kuFyBsth-1wuPqncnTiCa7reKzCz=RzO1X-Y78U74F5g@mail.gmail.com>
References: <CA+q6zcVjv1Lp-3=prBbpq2CbBioK91SHarfw3F8FHuUN4EwcUA@mail.gmail.com>
	<20180822175034.lnvwk5js32k6mjyz@alvherre.pgsql>
	<20180823041420.GA1158@paquier.xyz>
	<CAFh8B==4gpwor6ODercru0iifqwd0nQrdRvXzmorWkMRQa4q=g@mail.gmail.com>
	<CAFh8B=nHdsX83hTZMqGqbht39ud+DRQOv4_Te+xu3DuPssQbZA@mail.gmail.com>
	<CAFh8B=mYGP1J8ueHKU40Dr4bNBUPJj1rTbZ6Wpg4gR30uCRunA@mail.gmail.com>
	<20180829061723.GA5903@paquier.xyz>
	<CAFh8B=kuFyBsth-1wuPqncnTiCa7reKzCz=RzO1X-Y78U74F5g@mail.gmail.com>

On Wed, Aug 29, 2018 at 08:59:16AM +0200, Alexander Kukushkin wrote:
> Why the block 72478 of index relfile doesn't meet our expectations
> (contains so few tuples)?
> The answer to this question is in the page header. LSN, written in the
> indexpage header is AB3/56BF3B68.
> That has only one meaning, while the postgres was working before the
> crash it managed to apply WAL stream til at least AB3/56BF3B68, what
> is far ahead of "Minimum recovery ending location:     AB3/4A1B3118".

Yeah, that's the pinpoint.  Do you know by chance what was the content
of the control file for each standby you have upgraded to 9.6.10 before
starting them with the new binaries?  You mentioned a cluster of three
nodes, so I guess that you have two standbys, and that one of them did
not see the symptoms discussed here, while the other saw them.  Do you
still have the logs of the recovery just after starting the other
standby with 9.4.10 which did not see the symptom?  All your standbys
are using the background worker which would cause the btree deletion
code to be scanned, right?

I am trying to work on a reproducer with a bgworker starting once
recovery has been reached, without success yet.  Does your cluster
generate some XLOG_PARAMETER_CHANGE records?  In some cases, 9.4.8 could
have updated minRecoveryPoint to go backward, which is something that
8d68ee6 has been working on addressing.

Did you also try to use local WAL segments up where AB3/56BF3B68 is
applied, and also have a restore_command so as extra WAL segment fetches
from the archive would happen?
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

view thread (35+ messages)  latest in thread

Message-ID: <20180829121051.GC5903@paquier.xyz>
Permalink:  ../20180829121051.GC5903@paquier.xyz/
Also on:    postgresql.org/message-id/20180829121051.GC5903@paquier.xyz

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: michael@paquier.xyz, cyberdemn@gmail.com, alvherre@2ndquadrant.com, 9erthalion6@gmail.com
  Subject: Re: BUG #15346: Replica fails to start after the crash
  In-Reply-To: <20180829121051.GC5903@paquier.xyz>
  References: <CA+q6zcVjv1Lp-3=prBbpq2CbBioK91SHarfw3F8FHuUN4EwcUA@mail.gmail.com> <20180822175034.lnvwk5js32k6mjyz@alvherre.pgsql> <20180823041420.GA1158@paquier.xyz> <CAFh8B==4gpwor6ODercru0iifqwd0nQrdRvXzmorWkMRQa4q=g@mail.gmail.com> <CAFh8B=nHdsX83hTZMqGqbht39ud+DRQOv4_Te+xu3DuPssQbZA@mail.gmail.com> <CAFh8B=mYGP1J8ueHKU40Dr4bNBUPJj1rTbZ6Wpg4gR30uCRunA@mail.gmail.com> <20180829061723.GA5903@paquier.xyz> <CAFh8B=kuFyBsth-1wuPqncnTiCa7reKzCz=RzO1X-Y78U74F5g@mail.gmail.com> <20180829121051.GC5903@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox