Message-ID: <4EA900E5.9070905@enterprisedb.com>
Date: Thu, 27 Oct 2011 09:57:41 +0300
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI;
	rv:1.9.2.21) Gecko/20110831 Icedove/3.1.13
MIME-Version: 1.0
To: Florian Pflug <fgp@phlo.org>
CC: Simon Riggs <simon@2ndQuadrant.com>, Aidan Van Dyk <aidan@highrise.ca>,
	Daniel Farina <daniel@heroku.com>, Chris Redekop <chris@replicon.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Hot Backup with rsync fails at pg_clog if under load
References: 
 <CAC2SuRJTtsXL67EuTdd25GThYZBZTcepfG7Z0Vvv2pZQWzJD=w@mail.gmail.com>
	<CAC2SuRJPNqPe7Ga8LT6Q-vOgn05BmiH=y6F_dfiVu5u3NfOs=w@mail.gmail.com>
	<CAAZKuFbUSxOb8x80k9-fD9CsXBRmzTNWutqcDVGaTvwFTvURiQ@mail.gmail.com>
	<213AEFBF-479C-4F9E-B0CA-24AAB2C9F4DD@phlo.org>
	<CA+U5nMJksETN=sJEmu2FqhWNv2Uv63e_9T5OrZEGmviNnWEC5Q@mail.gmail.com>
	<CA+U5nMJfnaay7p_rRzrvWLTyO-aKNEh414czr3jyXT8vhHiJVw@mail.gmail.com>
	<1A94CE1A-C3E5-4923-B677-623D41BC273F@phlo.org>
	<CA+U5nMKUUoA8kRG=itfsO5nZuE3x_KDJz78EaUN3_FKmq-uMJA@mail.gmail.com>
	<6C3D7EDA-573E-46DC-9047-5FEB92876DA8@phlo.org>
	<CA+U5nMJLDZ7VP1O3Jdde+p+9-HxuEi8qG+v5+LuoZGSiOvhkxA@mail.gmail.com>
	<CAC_2qU_eCUYMY6O5CgAK0pYugd=moApirdAt8G_G2sYKs0DfOQ@mail.gmail.com>
	<CA+U5nML+fJgz-NdZrGX+R3m6JKnT1JC=v0iXkkSAsojnsqqPoQ@mail.gmail.com>
	<FD57812A-6A9B-44F3-8CDF-3CD2F8D9918E@phlo.org>
	<12E895D5-C6D0-4E71-B9C3-85E1BC14E6B8@phlo.org>
	<CA+U5nMLyW=S1DTBhswOirf_EZX06epvtyUZN364BGPJBDH
	_tiA@mail.gmail.com>
	<5EDA27FA-963F-413D-9849-874072F317E5@phlo.org>
In-Reply-To: <5EDA27FA-963F-413D-9849-874072F317E5@phlo.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 27.10.2011 02:29, Florian Pflug wrote:
> Per my theory about the cause of the problem in my other mail, I think you
> might see StartupCLOG failures even during crash recovery, provided that
> wal_level was set to hot_standby when the primary crashed. Here's how
>
> 1) We start a checkpoint, and get as far as LogStandbySnapshot()
> 2) A backend does AssignTransactionId, and gets as far as GetTransactionoId().
>    The assigned XID requires CLOG extension.
> 3) The checkpoint continues, and LogStandbySnapshot () advances the
>    checkpoint's nextXid to the XID assigned in (2).
> 4) We crash after writing the checkpoint record, but before the CLOG
>    extension makes it to the disk, and before any trace of the XID assigned
>    in (2) makes it to the xlog.
>
> Then StartupCLOG() would fail at the end of recovery, because we'd end up
> with a nextXid whose corresponding CLOG page doesn't exist.

No, clog extension is WAL-logged while holding the XidGenLock. At step 
3, LogStandbySnapshot() would block until the clog-extension record is 
written to WAL, so crash recovery would see and replay that record 
before calling StartupCLOG().

That can happen during hot standby, though, because StartupCLOG() is 
called earlier.

> My suggestion is to fix the CLOG problem in that same way that you fixed
> the SUBTRANS problem, i.e. by moving LogStandbySnapshot() to before
> CheckPointGuts().
>
> Here's what I image CreateCheckPoint() should look like:
>
> 1) LogStandbySnapshot() and fill out oldestActiveXid
> 2) Fill out REDO
> 3) Wait for concurrent commits
> 4) Fill out nextXid and the other fields
> 5) CheckPointGuts()
> 6) Rest
>
> It's then no longer necessary for LogStandbySnapshot() do modify
> the nextXid, since we fill out nextXid after LogStandbySnapshot() and
> will thus derive a higher value than LogStandbySnapshot() would have.

Hmm, I don't think that fully fixes the problem. Even if you're certain 
that CheckPointGuts() has fsync'd the clog page to disk, VACUUM might 
decide to truncate it away again while the checkpoint is running.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com