Subject: Incrementally Updated Backups and restartpoints

From: Heikki Linnakangas <[email-masked]>
Date: Wed, 13 Jan 2010 13:36:02 +0200

Our documentation suggests that you can take a base backup of a warm
standby server while it's running:

> If we take a backup of the standby server's data directory while it is processing logs shipped from the primary, we will be able to reload that data and restart the standby's recovery process from the last restart point. We no longer need to keep WAL files from before the restart point. If we need to recover, it will be faster to recover from the incrementally updated backup than from the original base backup. 

That doesn't seem safe. If the server makes a new restartpoint while the
backup is running, and pg_control is backed up after the new
restartpoint is made, recovery will restart from the new restartpoint.
That is wrong; recovery needs to restart at the restartpoint that was
most recent when the backup started. This is basically the same issue we
have solved in master with the backup_label file.

I wonder if it would be enough to document that pg_control must be
backed up first?

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

---

From: Fujii Masao <[email-masked]>
Date: Wed, 13 Jan 2010 20:57:37 +0900

> have solved in master with the backup_label file.

Right.

> I wonder if it would be enough to document that pg_control must be

Probably No. The archive recovery from such base backup would always
fail at the end of recovery because there is no backup-end record,
i.e., pg_stop_backup() is not executed in that case.

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

---

From: Heikki Linnakangas <[email-masked]>
Date: Wed, 13 Jan 2010 14:34:05 +0200

> i.e., pg_stop_backup() is not executed in that case.

No, that's not an issue. We only wait for the backup-end record if we
haven't seen yet since we started recovery from the base backup.
Assuming the standby had reached that point already before the new
backup from the standby started, backupStartLoc is zero in the control file.

---

From: Fujii Masao <[email-masked]>
Date: Thu, 14 Jan 2010 07:13:05 +0900

> backup from the standby started, backupStartLoc is zero in the control file.

OK. That assumption should be documented?

And, when we start an archive recovery from the backup from the standby,
we seem to reach a safe starting point before database has actually become
consistent. It's because backupStartLoc is zero. Isn't this an issue?

Regards,

---

From: Fujii Masao <[email-masked]>
Date: Thu, 4 Mar 2010 21:00:33 +0900

Hi,

I thought of this issue again since the related question arrived.
http://archives.postgresql.org/pgsql-admin/2010-03/msg00036.php

> OK. That assumption should be documented?

This comment is meaningless. Sorry for noise.

> consistent. It's because backupStartLoc is zero. Isn't this an issue?

This issue seems to still happen. So should this be fixed for 9.0?
Or only writing a note in document is enough for 9.0? I'm leaning
towards the latter.

Regards,

---

From: Fujii Masao <[email-masked]>
Date: Fri, 26 Mar 2010 22:21:42 +0900

> Or only writing a note in document is enough for 9.0? I'm leaning

I'm thinking of adding something like the following to the section
"25.6. Incrementally Updated Backups". Thought?

    The pg_control file must be backed up first.
    This avoids the problem that we might fail to restore a consistent
    database state because recovery starts from the later restart point
    than the start of the backup.

    When recovering from the incrementally updated backup, the server
    can begin accepting connections and complete the recovery successfully
    before the database has become consistent. To avoid these problems,
    you must check whether the database has been consistent by comparing
    the progress of the recovery with the backup ending WAL location
    before your users try to connect to the server and when archive
    recovery ends. So, in advance, the backup ending WAL location must
    be taken by calling the pg_last_xlog_replay_location function at the
    end of the backup. The progress of the recovery is also taken from
    the pg_last_xlog_replay_location function.

Regards,