public inbox for [email protected]
help / color / mirror / Atom feedRe: Recovery of .partial WAL segments
5+ messages / 4 participants
[nested] [flat]
* Re: Recovery of .partial WAL segments
@ 2024-08-09 14:29 Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Dmitry Dolgov @ 2024-08-09 14:29 UTC (permalink / raw)
To: Stefan Fercot <[email protected]>; +Cc: [email protected] <[email protected]>
> On Fri, Aug 02, 2024 at 08:47:02AM GMT, Stefan Fercot wrote:
>
> Not sure why CFbot CI fails on macOS/Windows while it works with the Github
> CI on my fork (
> https://cirrus-ci.com/github/pgstef/postgres/partial-walseg-recovery).
I guess it's because the test has to wait a bit after the node has been
started until the log lines will appear. One can see it in the
node_pitr3 logs, first it was hit by
SELECT pg_is_in_recovery() = 'f'
and only some moments later produced
restored log file "000000010000000000000003.partial" from archive
where the test has those operations in reversed order. Seems like the
retry loop from 019_replslot_limit might help.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Recovery of .partial WAL segments
@ 2024-08-09 18:25 Stefan Fercot <[email protected]>
parent: Dmitry Dolgov <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Fercot @ 2024-08-09 18:25 UTC (permalink / raw)
To: Dmitry Dolgov <[email protected]>; +Cc: [email protected] <[email protected]>
Hi,
On Fri, Aug 9, 2024 at 4:29 PM Dmitry Dolgov <[email protected]> wrote:
> Seems like the retry loop from 019_replslot_limit might help.
>
Thanks for the tip. Attached v2 adds the retry loop in the test which would
hopefully fix the cfbot.
Kind Regards,
Stefan
Attachments:
[text/x-patch] v2-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch (4.3K, ../../CAPzjO7X+0GG+sCn73tQughD+R+VHRdqj2cqGMk=UoC6h--=EXQ@mail.gmail.com/3-v2-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch)
download | inline diff:
From 203747394503b479338980948ccb3df84dc8d1a1 Mon Sep 17 00:00:00 2001
From: Stefan Fercot <[email protected]>
Date: Fri, 5 Apr 2024 10:57:03 +0200
Subject: [PATCH v2.] Make XLogFileRead try to restore .partial wal archives
Try to restore the normal archived wal segment first and, if not found, then try to restore the archived .partial wal segment.
This is safe because the next completed wal segment should contain at least the same data.
---
src/backend/access/transam/xlogrecovery.c | 20 +++++++++-
src/test/recovery/t/028_pitr_timelines.pl | 47 ++++++++++++++++++++++-
2 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 2ed3ea2b45..c8e793e3d9 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -4196,6 +4196,8 @@ XLogFileRead(XLogSegNo segno, int emode, TimeLineID tli,
char activitymsg[MAXFNAMELEN + 16];
char path[MAXPGPATH];
int fd;
+ char *partialxlogfname;
+ bool restoredArchivedFile;
XLogFileName(xlogfname, tli, segno, wal_segment_size);
@@ -4207,10 +4209,24 @@ XLogFileRead(XLogSegNo segno, int emode, TimeLineID tli,
xlogfname);
set_ps_display(activitymsg);
- if (!RestoreArchivedFile(path, xlogfname,
+ /*
+ * Try to restore the normal wal segment first and, if not found,
+ * then try to restore the .partial wal segment.
+ */
+
+ partialxlogfname = psprintf("%s.partial", xlogfname);
+
+ restoredArchivedFile = !RestoreArchivedFile(path, xlogfname,
+ "RECOVERYXLOG",
+ wal_segment_size,
+ InRedo) &&
+ !RestoreArchivedFile(path, partialxlogfname,
"RECOVERYXLOG",
wal_segment_size,
- InRedo))
+ InRedo);
+
+ pfree(partialxlogfname);
+ if (restoredArchivedFile)
return -1;
break;
diff --git a/src/test/recovery/t/028_pitr_timelines.pl b/src/test/recovery/t/028_pitr_timelines.pl
index 4b7d825b71..a958a0c6e7 100644
--- a/src/test/recovery/t/028_pitr_timelines.pl
+++ b/src/test/recovery/t/028_pitr_timelines.pl
@@ -110,7 +110,7 @@ $node_standby->stop;
# segment 000000020000000000000003, before the timeline switching
# record. (They are also present in the
# 000000010000000000000003.partial file, but .partial files are not
-# used automatically.)
+# used when recovering along the latest timeline by default.)
# Now test PITR to the recovery target. It should find the WAL in
# segment 000000020000000000000003, but not follow the timeline switch
@@ -173,4 +173,49 @@ $node_pitr2->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';")
$result = $node_pitr2->safe_psql('postgres', "SELECT max(i) FROM foo;");
is($result, qq{3}, "check table contents after point-in-time recovery");
+# The 000000010000000000000003.partial file could have been generated
+# by pg_receivewal without any standby node involved. In this case, we
+# wouldn't be able to recover from 000000020000000000000003.
+# Now, test PITR to the initial recovery target staying on the backup's
+# current timeline, trying to fetch the data from the
+# 000000010000000000000003.partial file.
+
+my $node_pitr3 = PostgreSQL::Test::Cluster->new('node_pitr3');
+$node_pitr3->init_from_backup(
+ $node_primary, $backup_name,
+ standby => 0,
+ has_restoring => 1);
+$node_pitr3->append_conf(
+ 'postgresql.conf', qq{
+recovery_target_name = 'rp'
+recovery_target_action = 'promote'
+recovery_target_timeline = 'current'
+});
+
+my $log_offset = -s $node_pitr3->logfile;
+$node_pitr3->start;
+
+my $msg_logged = 0;
+my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
+while ($max_attempts-- >= 0)
+{
+ if ($node_pitr3->log_contains(
+ "restored log file \"000000010000000000000003.partial\" from archive",
+ $log_offset))
+ {
+ $msg_logged = 1;
+ last;
+ }
+ sleep 1;
+}
+ok($msg_logged, "restored 000000010000000000000003.partial");
+
+# Wait until recovery finishes.
+$node_pitr3->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';")
+ or die "Timed out while waiting for PITR promotion";
+
+# Check that we see the data we expect.
+$result = $node_pitr3->safe_psql('postgres', "SELECT max(i) FROM foo;");
+is($result, qq{1}, "check table contents after point-in-time recovery");
+
done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Recovery of .partial WAL segments
@ 2024-10-18 09:07 Stepan Neretin <[email protected]>
parent: Stefan Fercot <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Stepan Neretin @ 2024-10-18 09:07 UTC (permalink / raw)
To: Stefan Fercot <[email protected]>; +Cc: Dmitry Dolgov <[email protected]>; [email protected] <[email protected]>
Hi there! It looks good to me! But I have a question. How is the
partialxlogfname freed in case of an error?
Best regards, Stepan Neretin.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Recovery of .partial WAL segments
@ 2024-10-18 12:39 Stefan Fercot <[email protected]>
parent: Stepan Neretin <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Stefan Fercot @ 2024-10-18 12:39 UTC (permalink / raw)
To: Stepan Neretin <[email protected]>; +Cc: Dmitry Dolgov <[email protected]>; [email protected] <[email protected]>
Hi,
On Fri, Oct 18, 2024 at 11:07 AM Stepan Neretin <[email protected]> wrote:
> Hi there! It looks good to me!
>
Thanks!
But I have a question. How is the partialxlogfname freed in case of an
> error?
I'm not sure I understand your question. What kind of error would you
expect exactly?
I mostly added a pfree there because we don't need it further. But why
would it be more of a problem compared to other variables/pointers?
Kind regards,
Stefan
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: PSQL - prevent describe listing tables that are already in listed schemas
@ 2026-07-10 09:51 Amit Kapila <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Amit Kapila @ 2026-07-10 09:51 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Peter Smith <[email protected]>; vignesh C <[email protected]>; Nisha Moond <[email protected]>; Jim Jones <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Jul 9, 2026 at 2:11 PM Shlok Kyal <[email protected]> wrote:
>
> On Thu, 9 Jul 2026 at 04:24, Peter Smith <[email protected]> wrote:
> >
> > On Wed, Jul 8, 2026 at 9:06 PM vignesh C <[email protected]> wrote:
> > >
> > > On Wed, 8 Jul 2026 at 04:58, Peter Smith <[email protected]> wrote:
> > > >
> > > > A rebase was needed.
> > > >
> > > > PSA v7.
> > >
> > > One minor comment:
> > > Can we add a comment here too, similar to how you have added for
> > > describe publications:
> >
> > Modified as suggested.
> >
> > PSA v8.
> >
>
> Hi Peter,
>
> I manually tested the patch, and it looks good to me.
>
The patch LGTM as well. I'll push this early next week.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-07-10 09:51 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-08-09 14:29 Re: Recovery of .partial WAL segments Dmitry Dolgov <[email protected]>
2024-08-09 18:25 ` Stefan Fercot <[email protected]>
2024-10-18 09:07 ` Stepan Neretin <[email protected]>
2024-10-18 12:39 ` Stefan Fercot <[email protected]>
2026-07-10 09:51 Re: PSQL - prevent describe listing tables that are already in listed schemas Amit Kapila <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox