Subject: Re: Re: [COMMITTERS] pgsql: Make standby server
	continuously retry restoring the next WAL
From: Simon Riggs <simon@2ndQuadrant.com>
To: Fujii Masao <masao.fujii@gmail.com>
Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>,
	Aidan Van Dyk <aidan@highrise.ca>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
In-Reply-To: <3f0b79eb1003241908n1e8f38e0q7cd7465163b3d7af@mail.gmail.com>
References: <3f0b79eb1002092105r21e009d3v468496058ba04392@mail.gmail.com>
	<4B743E7D.5070603@enterprisedb.com>
	<3f0b79eb1002180337t1fab1395ve3491256672af15f@mail.gmail.com>
	<4BA0B079.3050301@enterprisedb.com>
	<3f0b79eb1003180727g7877743eq81274e014fe70a49@mail.gmail.com>
	<1268988724.3556.3.camel@ebony> <4BA361E4.7020309@enterprisedb.com>
	<3f0b79eb1003230017v16f4ecbeyc20e75beeffe8f1c@mail.gmail.com>
	<4BAA060A.2020000@enterprisedb.com> <1269472981.8481.8946.camel@ebony>
	<3f0b79eb1003241908n1e8f38e0q7cd7465163b3d7af@mail.gmail.com>
Content-Type: text/plain
Date: Thu, 25 Mar 2010 08:08:11 +0000
Message-Id: <1269504491.8481.8965.camel@ebony>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Thu, 2010-03-25 at 11:08 +0900, Fujii Masao wrote:
> On Thu, Mar 25, 2010 at 8:23 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> > PANICing won't change the situation, so it just destroys server
> > availability. If we had 1 master and 42 slaves then this behaviour would
> > take down almost the whole server farm at once. Very uncool.
> >
> > You might have reason to prevent the server starting up at that point,
> > when in standby mode, but that is not a reason to PANIC. We don't really
> > want all of the standbys thinking they can be the master all at once
> > either. Better to throw a serious ERROR and have the server still up and
> > available for reads.
> 
> OK. How about making the startup process emit WARNING, stop WAL replay and
> wait for the presence of trigger file, when an invalid record is found?
> Which keeps the server up for readonly queries. And if the trigger file is
> found, I think that the startup process should emit a FATAL, i.e., the
> server should exit immediately, to prevent the server from becoming the
> primary in a half-finished state. Also to allow such a halfway failover,
> we should provide fast failover mode as pg_standby does?

The lack of docs begins to show a lack of coherent high-level design
here. By now, I've forgotten what this thread was even about. The major
design decision in this that keeps showing up is "remove pg_standby, at
all costs" but no reason has ever been given for that. I do believe
there is a "better way", but we won't find it by trial and error, even
if we had time to do so.

Please work on some clear docs for the failure modes in this system.
That way we can all read them and understand them, or point out further
issues. Moving straight to code is not a solution to this, since what we
need now is to all agree on the way forwards. If we ignore this, then
there is considerable risk that streaming rep will have a fatal
operational flaw.

Please just document/diagram how it works now, highlighting the problems
that still remain to be solved. We're all behind you and I'm helping
wherever I can.

-- 
 Simon Riggs           www.2ndQuadrant.com