pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
To: Florian Pflug <fgp@phlo.org>
Cc: Simon Riggs <simon@2ndQuadrant.com>
Cc: Aidan Van Dyk <aidan@highrise.ca>
Cc: Daniel Farina <daniel@heroku.com>
Cc: Chris Redekop <chris@replicon.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Hot Backup with rsync fails at pg_clog if under load
Date: Thu, 27 Oct 2011 09:57:41 +0300
Message-ID: <4EA900E5.9070905@enterprisedb.com> (raw)
In-Reply-To: <5EDA27FA-963F-413D-9849-874072F317E5@phlo.org>
References: <CAC2SuRJTtsXL67EuTdd25GThYZBZTcepfG7Z0Vvv2pZQWzJD=w@mail.gmail.com>
	<CAC2SuRJPNqPe7Ga8LT6Q-vOgn05BmiH=y6F_dfiVu5u3NfOs=w@mail.gmail.com>
	<CAAZKuFbUSxOb8x80k9-fD9CsXBRmzTNWutqcDVGaTvwFTvURiQ@mail.gmail.com>
	<213AEFBF-479C-4F9E-B0CA-24AAB2C9F4DD@phlo.org>
	<CA+U5nMJksETN=sJEmu2FqhWNv2Uv63e_9T5OrZEGmviNnWEC5Q@mail.gmail.com>
	<CA+U5nMJfnaay7p_rRzrvWLTyO-aKNEh414czr3jyXT8vhHiJVw@mail.gmail.com>
	<1A94CE1A-C3E5-4923-B677-623D41BC273F@phlo.org>
	<CA+U5nMKUUoA8kRG=itfsO5nZuE3x_KDJz78EaUN3_FKmq-uMJA@mail.gmail.com>
	<6C3D7EDA-573E-46DC-9047-5FEB92876DA8@phlo.org>
	<CA+U5nMJLDZ7VP1O3Jdde+p+9-HxuEi8qG+v5+LuoZGSiOvhkxA@mail.gmail.com>
	<CAC_2qU_eCUYMY6O5CgAK0pYugd=moApirdAt8G_G2sYKs0DfOQ@mail.gmail.com>
	<CA+U5nML+fJgz-NdZrGX+R3m6JKnT1JC=v0iXkkSAsojnsqqPoQ@mail.gmail.com>
	<FD57812A-6A9B-44F3-8CDF-3CD2F8D9918E@phlo.org>
	<12E895D5-C6D0-4E71-B9C3-85E1BC14E6B8@phlo.org>
	<CA+U5nMLyW=S1DTBhswOirf_EZX06epvtyUZN364BGPJBDH _tiA@mail.gmail.com>
	<5EDA27FA-963F-413D-9849-874072F317E5@phlo.org>

On 27.10.2011 02:29, Florian Pflug wrote:
> Per my theory about the cause of the problem in my other mail, I think you
> might see StartupCLOG failures even during crash recovery, provided that
> wal_level was set to hot_standby when the primary crashed. Here's how
>
> 1) We start a checkpoint, and get as far as LogStandbySnapshot()
> 2) A backend does AssignTransactionId, and gets as far as GetTransactionoId().
>    The assigned XID requires CLOG extension.
> 3) The checkpoint continues, and LogStandbySnapshot () advances the
>    checkpoint's nextXid to the XID assigned in (2).
> 4) We crash after writing the checkpoint record, but before the CLOG
>    extension makes it to the disk, and before any trace of the XID assigned
>    in (2) makes it to the xlog.
>
> Then StartupCLOG() would fail at the end of recovery, because we'd end up
> with a nextXid whose corresponding CLOG page doesn't exist.

No, clog extension is WAL-logged while holding the XidGenLock. At step 
3, LogStandbySnapshot() would block until the clog-extension record is 
written to WAL, so crash recovery would see and replay that record 
before calling StartupCLOG().

That can happen during hot standby, though, because StartupCLOG() is 
called earlier.

> My suggestion is to fix the CLOG problem in that same way that you fixed
> the SUBTRANS problem, i.e. by moving LogStandbySnapshot() to before
> CheckPointGuts().
>
> Here's what I image CreateCheckPoint() should look like:
>
> 1) LogStandbySnapshot() and fill out oldestActiveXid
> 2) Fill out REDO
> 3) Wait for concurrent commits
> 4) Fill out nextXid and the other fields
> 5) CheckPointGuts()
> 6) Rest
>
> It's then no longer necessary for LogStandbySnapshot() do modify
> the nextXid, since we fill out nextXid after LogStandbySnapshot() and
> will thus derive a higher value than LogStandbySnapshot() would have.

Hmm, I don't think that fully fixes the problem. Even if you're certain 
that CheckPointGuts() has fsync'd the clog page to disk, VACUUM might 
decide to truncate it away again while the checkpoint is running.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com



view thread (80+ messages)  latest in thread

Message-ID: <4EA900E5.9070905@enterprisedb.com>
Permalink:  ../4EA900E5.9070905@enterprisedb.com/
Also on:    postgresql.org/message-id/4EA900E5.9070905@enterprisedb.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: heikki.linnakangas@enterprisedb.com, fgp@phlo.org, simon@2ndQuadrant.com, aidan@highrise.ca, daniel@heroku.com, chris@replicon.com
  Subject: Re: Hot Backup with rsync fails at pg_clog if under load
  In-Reply-To: <4EA900E5.9070905@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox