public inbox for [email protected]
help / color / mirror / Atom feedIncremental backup from a streaming replication standby
9+ messages / 4 participants
[nested] [flat]
* Incremental backup from a streaming replication standby
@ 2024-06-29 05:01 Laurenz Albe <[email protected]>
0 siblings, 2 replies; 9+ messages in thread
From: Laurenz Albe @ 2024-06-29 05:01 UTC (permalink / raw)
To: pgsql-hackers
I played around with incremental backup yesterday and tried $subject
The WAL summarizer is running on the standby server, but when I try
to take an incremental backup, I get an error that I understand to mean
that WAL summarizing hasn't caught up yet.
I am not sure if that is working as designed, but if it is, I think it
should be documented.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-15 15:26 Laurenz Albe <[email protected]>
parent: Laurenz Albe <[email protected]>
1 sibling, 1 reply; 9+ messages in thread
From: Laurenz Albe @ 2024-07-15 15:26 UTC (permalink / raw)
To: pgsql-hackers
On Sat, 2024-06-29 at 07:01 +0200, Laurenz Albe wrote:
> I played around with incremental backup yesterday and tried $subject
>
> The WAL summarizer is running on the standby server, but when I try
> to take an incremental backup, I get an error that I understand to mean
> that WAL summarizing hasn't caught up yet.
>
> I am not sure if that is working as designed, but if it is, I think it
> should be documented.
I played with this some more. Here is the exact error message:
ERROR: manifest requires WAL from final timeline 1 ending at 0/1967C260, but this backup starts at 0/1967C190
By trial and error I found that when I run a CHECKPOINT on the primary,
taking an incremental backup on the standby works.
I couldn't fathom the cause of that, but I think that that should either
be addressed or documented before v17 comes out.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 14:52 Robert Haas <[email protected]>
parent: Laurenz Albe <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: Robert Haas @ 2024-07-19 14:52 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: pgsql-hackers
On Mon, Jul 15, 2024 at 11:27 AM Laurenz Albe <[email protected]> wrote:
> On Sat, 2024-06-29 at 07:01 +0200, Laurenz Albe wrote:
> > I played around with incremental backup yesterday and tried $subject
> >
> > The WAL summarizer is running on the standby server, but when I try
> > to take an incremental backup, I get an error that I understand to mean
> > that WAL summarizing hasn't caught up yet.
> >
> > I am not sure if that is working as designed, but if it is, I think it
> > should be documented.
>
> I played with this some more. Here is the exact error message:
>
> ERROR: manifest requires WAL from final timeline 1 ending at 0/1967C260, but this backup starts at 0/1967C190
>
> By trial and error I found that when I run a CHECKPOINT on the primary,
> taking an incremental backup on the standby works.
>
> I couldn't fathom the cause of that, but I think that that should either
> be addressed or documented before v17 comes out.
I had a feeling this was going to be confusing. I'm not sure what to
do about it, but I'm open to suggestions.
Suppose you take a full backup F; replay of that backup will begin
with a checkpoint CF. Then you try to take an incremental backup I;
replay will begin from a checkpoint CI. For the incremental backup to
be valid, it must include all blocks modified after CF and before CI.
But when the backup is taken on a standby, no new checkpoint is
possible. Hence, CI will be the most recent restartpoint on the
standby that has occurred before the backup starts. So, if F is taken
on the primary and then I is immediately taken on the standby without
the standby having done a new restartpoint, or if both F and I are
taken on the standby and no restartpoint intervenes, then CF=CI. In
that scenario, an incremental backup is pretty much pointless: every
single incremental file would contain 0 blocks. You might as well just
use the backup you already have, unless one of the non-relation files
has changed. So, except in that unusual corner case, the fact that the
backup fails isn't really costing you anything. In fact, there's a
decent chance that it's saving you from taking a completely useless
backup.
On the primary, this doesn't occur, because there, each new backup
triggers a new checkpoint, so you always have CI>CF.
The error message is definitely confusing. The reason I'm not sure how
to do better is that there is a large class of errors that a user
could make that would trigger an error of this general type. I'm
guessing that attempting a standby backup with CF=CI will turn out to
be the most common one, but I don't think it'll be the only one that
ever comes up. The code in PrepareForIncrementalBackup() focuses on
what has gone wrong on a technical level rather than on what you
probably did to create that situation. Indeed, the server doesn't
really know what you did to create that situation. You could trigger
the same error by taking a full backup on the primary and then try to
take an incremental based on that full backup on a time-delayed
standby (or a lagging standby) whose replay position was behind the
primary, i.e. CI<CF.
More perversely, you could trigger the error by spinning up a standby,
promoting it, taking a full backup, destroying the standby, removing
the timeline history file from the archive, spinning up a new standby,
promoting onto the same timeline ID as the previous one, and then
trying to take an incremental backup relative to the full backup. This
might actually succeed, if you take the incremental backup at a later
LSN than the previous full backup, but, as you may guess, terrible
things will happen to you if you try to use such a backup. (I hope you
will agree that this would be a self-inflicted injury; I can't see any
way of detecting such cases.) If the incremental backup LSN is earlier
than the previous full backup LSN, this error will trigger.
So, given all the above, what can we do here?
One option might be to add an errhint() to the message. I had trouble
thinking of something that was compact enough to be reasonable to
include and yet reasonably accurate and useful, but maybe we can
brainstorm and figure something out. Another option might be to add
more to the documentation, but it's all so complicated that I'm not
sure what to write. It feels hard to make something that is brief
enough to be worth including, accurate enough to help more than it
hurts, and understandable enough that people who run into this will be
able to make use of it.
I think I'm a little too close to this to really know what the best
thing to do is, so I'm happy to hear suggestions from you and others.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 15:32 David Steele <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: David Steele @ 2024-07-19 15:32 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Laurenz Albe <[email protected]>; +Cc: pgsql-hackers
On 7/19/24 21:52, Robert Haas wrote:
> On Mon, Jul 15, 2024 at 11:27 AM Laurenz Albe <[email protected]> wrote:
>> On Sat, 2024-06-29 at 07:01 +0200, Laurenz Albe wrote:
>>> I played around with incremental backup yesterday and tried $subject
>>>
>>> The WAL summarizer is running on the standby server, but when I try
>>> to take an incremental backup, I get an error that I understand to mean
>>> that WAL summarizing hasn't caught up yet.
>>>
>>> I am not sure if that is working as designed, but if it is, I think it
>>> should be documented.
>>
>> I played with this some more. Here is the exact error message:
>>
>> ERROR: manifest requires WAL from final timeline 1 ending at 0/1967C260, but this backup starts at 0/1967C190
>>
>> By trial and error I found that when I run a CHECKPOINT on the primary,
>> taking an incremental backup on the standby works.
>>
>> I couldn't fathom the cause of that, but I think that that should either
>> be addressed or documented before v17 comes out.
>
> I had a feeling this was going to be confusing. I'm not sure what to
> do about it, but I'm open to suggestions.
>
> Suppose you take a full backup F; replay of that backup will begin
> with a checkpoint CF. Then you try to take an incremental backup I;
> replay will begin from a checkpoint CI. For the incremental backup to
> be valid, it must include all blocks modified after CF and before CI.
> But when the backup is taken on a standby, no new checkpoint is
> possible. Hence, CI will be the most recent restartpoint on the
> standby that has occurred before the backup starts. So, if F is taken
> on the primary and then I is immediately taken on the standby without
> the standby having done a new restartpoint, or if both F and I are
> taken on the standby and no restartpoint intervenes, then CF=CI. In
> that scenario, an incremental backup is pretty much pointless: every
> single incremental file would contain 0 blocks. You might as well just
> use the backup you already have, unless one of the non-relation files
> has changed. So, except in that unusual corner case, the fact that the
> backup fails isn't really costing you anything. In fact, there's a
> decent chance that it's saving you from taking a completely useless
> backup.
<snip>
> I think I'm a little too close to this to really know what the best
> thing to do is, so I'm happy to hear suggestions from you and others.
I think it would be enough just to add a hint such as:
HINT: this is possible when making a standby backup with little or no
activity.
My guess is in production environments this will be uncommon.
For example, over the years we (pgBackRest) have gotten numerous bug
reports that time-targeted PITR does not work. In every case we found
that the user was just testing procedures and the database had no
activity between backups -- therefore recovery had no commit timestamps
to use to end recovery. Test environments sometimes produce weird results.
Having said that, I think it would be better if it worked even if it
does produce an empty backup. An empty backup wastes some disk space but
if it produces less friction and saves an admin having to intervene then
it is probably worth it. I don't immediately see how to do that in a
reliable way, though, and in any case it seems like something to
consider for PG18.
Regards,
-David
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 16:59 Robert Haas <[email protected]>
parent: David Steele <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: Robert Haas @ 2024-07-19 16:59 UTC (permalink / raw)
To: David Steele <[email protected]>; +Cc: Laurenz Albe <[email protected]>; pgsql-hackers
On Fri, Jul 19, 2024 at 11:32 AM David Steele <[email protected]> wrote:
> I think it would be enough just to add a hint such as:
>
> HINT: this is possible when making a standby backup with little or no
> activity.
That could work (with "this" capitalized).
> My guess is in production environments this will be uncommon.
I think so too, but when it does happen, confusion may be common.
> Having said that, I think it would be better if it worked even if it
> does produce an empty backup. An empty backup wastes some disk space but
> if it produces less friction and saves an admin having to intervene then
> it is probably worth it. I don't immediately see how to do that in a
> reliable way, though, and in any case it seems like something to
> consider for PG18.
Yeah, I'm pretty reluctant to weaken the sanity checks here, at least
in the short term. Note that what the check is actually complaining
about is that the previous backup thinks that the WAL it needs to
replay to reach consistency ends after the start of the current
backup. Even in this scenario, I'm not positive that everything would
be OK if we let the backup proceed, and it's easy to think of
scenarios where it definitely isn't. Plus, it's not quite clear how to
distinguish the cases where it's OK from the cases where it isn't.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 18:41 Laurenz Albe <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: Laurenz Albe @ 2024-07-19 18:41 UTC (permalink / raw)
To: Robert Haas <[email protected]>; David Steele <[email protected]>; +Cc: pgsql-hackers
On Fri, 2024-07-19 at 12:59 -0400, Robert Haas wrote:
Thanks for looking at this.
> On Fri, Jul 19, 2024 at 11:32 AM David Steele <[email protected]> wrote:
> > I think it would be enough just to add a hint such as:
> >
> > HINT: this is possible when making a standby backup with little or no
> > activity.
>
> That could work (with "this" capitalized).
>
> > My guess is in production environments this will be uncommon.
>
> I think so too, but when it does happen, confusion may be common.
I guess this will most likely happen during tests like the one I made.
I'd be alright with the hint, but I'd say "during making an *incremental*
standby backup", because that's the only case where it can happen.
I think it would also be sufficient if we document that possibility.
When I got the error, I looked at the documentation of incremental
backup for any limitations with standby servers, but didn't find any.
A remark in the documentation would have satisfied me.
Yours,
Laurenz
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 20:03 Robert Haas <[email protected]>
parent: Laurenz Albe <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: Robert Haas @ 2024-07-19 20:03 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: David Steele <[email protected]>; pgsql-hackers
On Fri, Jul 19, 2024 at 2:41 PM Laurenz Albe <[email protected]> wrote:
> I'd be alright with the hint, but I'd say "during making an *incremental*
> standby backup", because that's the only case where it can happen.
>
> I think it would also be sufficient if we document that possibility.
> When I got the error, I looked at the documentation of incremental
> backup for any limitations with standby servers, but didn't find any.
> A remark in the documentation would have satisfied me.
Would you like to propose a patch adding a hint and/or adjusting the
documentation? Or are you wanting me to do that?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby fails
@ 2024-07-19 22:07 Laurenz Albe <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Laurenz Albe @ 2024-07-19 22:07 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: David Steele <[email protected]>; pgsql-hackers
On Fri, 2024-07-19 at 16:03 -0400, Robert Haas wrote:
> On Fri, Jul 19, 2024 at 2:41 PM Laurenz Albe <[email protected]> wrote:
> > I'd be alright with the hint, but I'd say "during making an *incremental*
> > standby backup", because that's the only case where it can happen.
> >
> > I think it would also be sufficient if we document that possibility.
> > When I got the error, I looked at the documentation of incremental
> > backup for any limitations with standby servers, but didn't find any.
> > A remark in the documentation would have satisfied me.
>
> Would you like to propose a patch adding a hint and/or adjusting the
> documentation? Or are you wanting me to do that?
Here is a patch.
I went for both the errhint and some documentation.
Yours,
Laurenz Albe
Attachments:
[text/x-patch] v1-0001-Add-documentation-and-hint-for-incremental-backup.patch (2.6K, ../../[email protected]/2-v1-0001-Add-documentation-and-hint-for-incremental-backup.patch)
download | inline diff:
From a46b23afc919bfa0d0aa9bdad962b95a3ae407ec Mon Sep 17 00:00:00 2001
From: Laurenz Albe <[email protected]>
Date: Sat, 20 Jul 2024 00:05:26 +0200
Subject: [PATCH v1] Add documentation and hint for incremental backup on
standbys
Taking an incremental backup on a streaming replication standby immediately
after a base backup can result in the error
manifest requires WAL from final timeline n ending at XXX, but this backup starts at YYY
This message looks scary, even though the only problem is that the backup
would be empty and thus it makes no sense to take it anyway.
Add a clarifying errhint and some documentation to mitigate the problem.
Author: Laurenz Albe
Reviewed-by: Robert Haas, David Steele
Discussion: https://postgr.es/m/[email protected]
Backpatch to v17.
---
doc/src/sgml/backup.sgml | 9 +++++++++
src/backend/backup/basebackup_incremental.c | 3 ++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 91da3c26ba..0f84ebc36e 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -925,6 +925,15 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
to manage. For a large database all of which is heavily modified,
incremental backups won't be much smaller than full backups.
</para>
+
+ <para>
+ Like a base backup, you can take an incremental backup from a streaming
+ replication standby server. But since a backup of a standby server cannot
+ initiate a checkpoint, it is possible that an incremental backup taken
+ right after a base backup will fail with an error, since it would have
+ to start with the same checkpoint as the base backup and would therefore
+ be empty.
+ </para>
</sect2>
<sect2 id="backup-lowlevel-base-backup">
diff --git a/src/backend/backup/basebackup_incremental.c b/src/backend/backup/basebackup_incremental.c
index 2108702397..85c16182a6 100644
--- a/src/backend/backup/basebackup_incremental.c
+++ b/src/backend/backup/basebackup_incremental.c
@@ -441,7 +441,8 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
errmsg("manifest requires WAL from final timeline %u ending at %X/%X, but this backup starts at %X/%X",
range->tli,
LSN_FORMAT_ARGS(range->end_lsn),
- LSN_FORMAT_ARGS(backup_state->startpoint))));
+ LSN_FORMAT_ARGS(backup_state->startpoint)),
+ errhint("Perhaps there was too little activity since the previous backup, and this incremental backup would be empty.")));
}
else
{
--
2.45.2
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Incremental backup from a streaming replication standby
@ 2024-07-22 06:05 Michael Paquier <[email protected]>
parent: Laurenz Albe <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: Michael Paquier @ 2024-07-22 06:05 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: pgsql-hackers
On Sat, Jun 29, 2024 at 07:01:04AM +0200, Laurenz Albe wrote:
> The WAL summarizer is running on the standby server, but when I try
> to take an incremental backup, I get an error that I understand to mean
> that WAL summarizing hasn't caught up yet.
Added an open item for this one.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2024-07-22 06:05 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-06-29 05:01 Incremental backup from a streaming replication standby Laurenz Albe <[email protected]>
2024-07-15 15:26 ` Re: Incremental backup from a streaming replication standby fails Laurenz Albe <[email protected]>
2024-07-19 14:52 ` Re: Incremental backup from a streaming replication standby fails Robert Haas <[email protected]>
2024-07-19 15:32 ` Re: Incremental backup from a streaming replication standby fails David Steele <[email protected]>
2024-07-19 16:59 ` Re: Incremental backup from a streaming replication standby fails Robert Haas <[email protected]>
2024-07-19 18:41 ` Re: Incremental backup from a streaming replication standby fails Laurenz Albe <[email protected]>
2024-07-19 20:03 ` Re: Incremental backup from a streaming replication standby fails Robert Haas <[email protected]>
2024-07-19 22:07 ` Re: Incremental backup from a streaming replication standby fails Laurenz Albe <[email protected]>
2024-07-22 06:05 ` Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox