DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	b=swQfz3Abz/mDTFjxYCz6Pt5i4uRvFY1C4422ArDmS8znMOrvsH84dj5hBapLWtkT1T
	Hef2344haR4eYGmWDyhpMtAzgY1R5ArKRDOps21mI7Ivwz3QQ27/pN7HdVcpThbjWuf9
	jr4qoxaWZnmgkZAxTD4CEvJb/IRCZpaY7FsoA=
MIME-Version: 1.0
In-Reply-To: <4BA0B079.3050301@enterprisedb.com>
References: <3f0b79eb1002092105r21e009d3v468496058ba04392@mail.gmail.com>
	<4B73FB99.4080403@enterprisedb.com> <1265893599.7341.1454.camel@ebony>
	<4B740613.5090004@enterprisedb.com>
	<20100211140118.GB14128@oak.highrise.ca>
	<4B74118C.30704@enterprisedb.com>
	<20100211144204.GC14128@oak.highrise.ca>
	<4B743E7D.5070603@enterprisedb.com>
	<3f0b79eb1002180337t1fab1395ve3491256672af15f@mail.gmail.com>
	<4BA0B079.3050301@enterprisedb.com>
Date: Thu, 18 Mar 2010 23:27:59 +0900
Message-ID: <3f0b79eb1003180727g7877743eq81274e014fe70a49@mail.gmail.com>
Subject: Re: Re: [COMMITTERS] pgsql: Make standby server
	continuously retry restoring the next WAL
From: Fujii Masao <masao.fujii@gmail.com>
To: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
Cc: Aidan Van Dyk <aidan@highrise.ca>, Simon Riggs <simon@2ndquadrant.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Mar 17, 2010 at 7:35 PM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> Fujii Masao wrote:
>> I found another missing feature in new file-based log shipping (i.e.,
>> standby_mode is enabled and 'cp' is used as restore_command).
>>
>> After the trigger file is found, the startup process with pg_standby
>> tries to replay all of the WAL files in both pg_xlog and the archive.
>> So, when the primary fails, if the latest WAL file in pg_xlog of the
>> primary can be read, we can prevent the data loss by copying it to
>> pg_xlog of the standby before creating the trigger file.
>>
>> On the other hand, the startup process with standby mode doesn't
>> replay the WAL files in pg_xlog after the trigger file is found. So
>> failover always causes the data loss even if the latest WAL file can
>> be read from the primary. And if the latest WAL file is copied to the
>> archive instead, it can be replayed but a PANIC error would happen
>> because it's not filled.
>>
>> We should remove this restriction?
>
> Looking into this, I realized that we have a bigger problem related to
> this. Although streaming replication stores the streamed WAL files in
> pg_xlog, so that they can be re-replayed after a standby restart without
> connecting to the master, we don't try to replay those either. So if you
> restart standby, it will fail to start up if the WAL it needs can't be
> found in archive or by connecting to the master. That must be fixed.

I agree that this is a bigger problem. Since the standby always starts
walreceiver before replaying any WAL files in pg_xlog, walreceiver tries
to receive the WAL files following the REDO starting point even if they
have already been in pg_xlog. IOW, the same WAL files might be shipped
from the primary to the standby many times. This behavior is unsmart,
and should be addressed.

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center