Subject: Re: Re: [COMMITTERS] pgsql: Make standby server
	continuously retry restoring the next WAL
From: Simon Riggs <simon@2ndQuadrant.com>
To: Fujii Masao <masao.fujii@gmail.com>
Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>,
	Aidan Van Dyk <aidan@highrise.ca>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
In-Reply-To: <3f0b79eb1003241908n1e8f38e0q7cd7465163b3d7af@mail.gmail.com>
References: <3f0b79eb1002092105r21e009d3v468496058ba04392@mail.gmail.com>
	<4B743E7D.5070603@enterprisedb.com>
	<3f0b79eb1002180337t1fab1395ve3491256672af15f@mail.gmail.com>
	<4BA0B079.3050301@enterprisedb.com>
	<3f0b79eb1003180727g7877743eq81274e014fe70a49@mail.gmail.com>
	<1268988724.3556.3.camel@ebony> <4BA361E4.7020309@enterprisedb.com>
	<3f0b79eb1003230017v16f4ecbeyc20e75beeffe8f1c@mail.gmail.com>
	<4BAA060A.2020000@enterprisedb.com> <1269472981.8481.8946.camel@ebony>
	<3f0b79eb1003241908n1e8f38e0q7cd7465163b3d7af@mail.gmail.com>
Content-Type: text/plain
Date: Thu, 25 Mar 2010 08:22:55 +0000
Message-Id: <1269505375.8481.8977.camel@ebony>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Thu, 2010-03-25 at 11:08 +0900, Fujii Masao wrote:
> On Thu, Mar 25, 2010 at 8:23 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> > PANICing won't change the situation, so it just destroys server
> > availability. If we had 1 master and 42 slaves then this behaviour would
> > take down almost the whole server farm at once. Very uncool.
> >
> > You might have reason to prevent the server starting up at that point,
> > when in standby mode, but that is not a reason to PANIC. We don't really
> > want all of the standbys thinking they can be the master all at once
> > either. Better to throw a serious ERROR and have the server still up and
> > available for reads.
> 
> OK. How about making the startup process emit WARNING, stop WAL replay and
> wait for the presence of trigger file, when an invalid record is found?
> Which keeps the server up for readonly queries. 

Yes. Receiving new WAL records is a completely separate activity from
running the rest of the server (in this release...).

> And if the trigger file is
> found, I think that the startup process should emit a FATAL, i.e., the
> server should exit immediately, to prevent the server from becoming the
> primary in a half-finished state. 

Please remember that "half-finished" is your judgment on what has
happened in the particular scenario you are considering. In many cases,
an invalid WAL record clearly and simply indicates the end of WAL and we
should start up normally.

"State" is a good word here. I'd like to see the server have a clear
state model with well documented transitions between them. The state
should also be externally queriable, so we can work out what its doing
and how long we can expect it to keep doing it for.

I don't want to be in a position where we are waiting for the server to
sort itself out from a complex set of retries. 

> Also to allow such a halfway failover,
> we should provide fast failover mode as pg_standby does?

Yes, we definitely need a JFDI solution for immediate failover.

-- 
 Simon Riggs           www.2ndQuadrant.com