public inbox for [email protected]help / color / mirror / Atom feed
Recovery of .partial WAL segments 4+ messages / 4 participants [nested] [flat]
* Recovery of .partial WAL segments @ 2024-04-05 09:45 Stefan Fercot <[email protected]> 2024-08-02 06:47 ` Re: Recovery of .partial WAL segments Stefan Fercot <[email protected]> 2024-10-18 13:56 ` Re: Recovery of .partial WAL segments Matthias van de Meent <[email protected]> 0 siblings, 2 replies; 4+ messages in thread From: Stefan Fercot @ 2024-04-05 09:45 UTC (permalink / raw) To: [email protected] <[email protected]> Dear hackers, Generating a ".partial" WAL segment is pretty common nowadays (using pg_receivewal or during standby promotion). However, we currently don't do anything with it unless the user manually removes that ".partial" extension. The 028_pitr_timelines tests are highlighting that fact: with test data being being in 000000020000000000000003 and 000000010000000000000003.partial, a recovery following the latest timeline (2) will succeed but fail if we follow the current timeline (1). By simply trying to fetch the ".partial" file in XLogFileRead, we can easily recover more data and also cover that (current timeline) recovery case. So, this proposed patch makes XLogFileRead try to restore ".partial" WAL archives and adds a test to 028_pitr_timelines using current recovery_target_timeline. As far as I've seen, the current pg_receivewal tests only seem to cover the archives generation but not actually trying to recover using it. I wasn't sure it was interesting to add such tests right now, so I didn't considered it for this patch. Many thanks in advance for your feedback and thoughts about this, Kind Regards, -- Stefan FERCOT Data Egret (https://dataegret.com) Attachments: [text/x-patch] v1-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch (4.1K, ../../PUrgXj3toL_CVxytc7fzYQNlY5FiUllklbS9UL1wU5ptiJAC2ChrF64EwO8neCKU9QnET5Zjv1HpMLFAgQPJQxOzM9f49jjkV56gEpUOYw8=@protonmail.com/2-v1-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch) download | inline diff: From 8c73284b72120ddf3537e1632f12455502d47f6d Mon Sep 17 00:00:00 2001 From: Stefan Fercot <[email protected]> Date: Fri, 5 Apr 2024 10:57:03 +0200 Subject: [PATCH v1.] Make XLogFileRead try to restore .partial wal archives Try to restore the normal archived wal segment first and, if not found, then try to restore the archived .partial wal segment. This is safe because the next completed wal segment should contain at least the same data. --- src/backend/access/transam/xlogrecovery.c | 20 ++++++++++-- src/test/recovery/t/028_pitr_timelines.pl | 37 ++++++++++++++++++++++- 2 files changed, 54 insertions(+), 3 deletions(-) diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index b2fe2d04cc..6d51b6a296 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -4206,6 +4206,8 @@ XLogFileRead(XLogSegNo segno, int emode, TimeLineID tli, char activitymsg[MAXFNAMELEN + 16]; char path[MAXPGPATH]; int fd; + char *partialxlogfname; + bool restoredArchivedFile; XLogFileName(xlogfname, tli, segno, wal_segment_size); @@ -4217,10 +4219,24 @@ XLogFileRead(XLogSegNo segno, int emode, TimeLineID tli, xlogfname); set_ps_display(activitymsg); - if (!RestoreArchivedFile(path, xlogfname, + /* + * Try to restore the normal wal segment first and, if not found, + * then try to restore the .partial wal segment. + */ + + partialxlogfname = psprintf("%s.partial", xlogfname); + + restoredArchivedFile = !RestoreArchivedFile(path, xlogfname, + "RECOVERYXLOG", + wal_segment_size, + InRedo) && + !RestoreArchivedFile(path, partialxlogfname, "RECOVERYXLOG", wal_segment_size, - InRedo)) + InRedo); + + pfree(partialxlogfname); + if (restoredArchivedFile) return -1; break; diff --git a/src/test/recovery/t/028_pitr_timelines.pl b/src/test/recovery/t/028_pitr_timelines.pl index 4b7d825b71..512695cd0a 100644 --- a/src/test/recovery/t/028_pitr_timelines.pl +++ b/src/test/recovery/t/028_pitr_timelines.pl @@ -110,7 +110,7 @@ $node_standby->stop; # segment 000000020000000000000003, before the timeline switching # record. (They are also present in the # 000000010000000000000003.partial file, but .partial files are not -# used automatically.) +# used when recovering along the latest timeline by default.) # Now test PITR to the recovery target. It should find the WAL in # segment 000000020000000000000003, but not follow the timeline switch @@ -173,4 +173,39 @@ $node_pitr2->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';") $result = $node_pitr2->safe_psql('postgres', "SELECT max(i) FROM foo;"); is($result, qq{3}, "check table contents after point-in-time recovery"); +# The 000000010000000000000003.partial file could have been generated +# by pg_receivewal without any standby node involved. In this case, we +# wouldn't be able to recover from 000000020000000000000003. +# Now, test PITR to the initial recovery target staying on the backup's +# current timeline, trying to fetch the data from the +# 000000010000000000000003.partial file. + +my $node_pitr3 = PostgreSQL::Test::Cluster->new('node_pitr3'); +$node_pitr3->init_from_backup( + $node_primary, $backup_name, + standby => 0, + has_restoring => 1); +$node_pitr3->append_conf( + 'postgresql.conf', qq{ +recovery_target_name = 'rp' +recovery_target_action = 'promote' +recovery_target_timeline = 'current' +}); + +my $log_offset = -s $node_pitr3->logfile; +$node_pitr3->start; + +ok( $node_pitr3->log_contains( + "restored log file \"000000010000000000000003.partial\" from archive", + $log_offset), + "restored 000000010000000000000003.partial"); + +# Wait until recovery finishes. +$node_pitr3->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';") + or die "Timed out while waiting for PITR promotion"; + +# Check that we see the data we expect. +$result = $node_pitr3->safe_psql('postgres', "SELECT max(i) FROM foo;"); +is($result, qq{1}, "check table contents after point-in-time recovery"); + done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Recovery of .partial WAL segments 2024-04-05 09:45 Recovery of .partial WAL segments Stefan Fercot <[email protected]> @ 2024-08-02 06:47 ` Stefan Fercot <[email protected]> 1 sibling, 0 replies; 4+ messages in thread From: Stefan Fercot @ 2024-08-02 06:47 UTC (permalink / raw) To: [email protected] <[email protected]> Hi, I've added a CF entry for this patch: https://commitfest.postgresql.org/49/5148/ Not sure why CFbot CI fails on macOS/Windows while it works with the Github CI on my fork ( https://cirrus-ci.com/github/pgstef/postgres/partial-walseg-recovery). Many thanks in advance for your feedback and thoughts about this patch, Kind Regards, -- Stefan FERCOT Data Egret (https://dataegret.com) On Thu, Aug 1, 2024 at 10:23 PM Stefan Fercot <[email protected]> wrote: > Dear hackers, > > Generating a ".partial" WAL segment is pretty common nowadays (using > pg_receivewal or during standby promotion). > However, we currently don't do anything with it unless the user manually > removes that ".partial" extension. > > The 028_pitr_timelines tests are highlighting that fact: with test data > being being in 000000020000000000000003 and > 000000010000000000000003.partial, a recovery following the latest timeline > (2) will succeed but fail if we follow the current timeline (1). > > By simply trying to fetch the ".partial" file in XLogFileRead, we can > easily recover more data and also cover that (current timeline) recovery > case. > > So, this proposed patch makes XLogFileRead try to restore ".partial" WAL > archives and adds a test to 028_pitr_timelines using current > recovery_target_timeline. > > As far as I've seen, the current pg_receivewal tests only seem to cover > the archives generation but not actually trying to recover using it. I > wasn't sure it was interesting to add such tests right now, so I didn't > considered it for this patch. > > Many thanks in advance for your feedback and thoughts about this, > Kind Regards, > -- > Stefan FERCOT > Data Egret (https://dataegret.com) ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Recovery of .partial WAL segments 2024-04-05 09:45 Recovery of .partial WAL segments Stefan Fercot <[email protected]> @ 2024-10-18 13:56 ` Matthias van de Meent <[email protected]> 1 sibling, 0 replies; 4+ messages in thread From: Matthias van de Meent @ 2024-10-18 13:56 UTC (permalink / raw) To: Stefan Fercot <[email protected]>; +Cc: [email protected] <[email protected]> On Fri, 5 Apr 2024 at 11:45, Stefan Fercot <[email protected]> wrote: > > Dear hackers, > > Generating a ".partial" WAL segment is pretty common nowadays (using pg_receivewal or during standby promotion). > However, we currently don't do anything with it unless the user manually removes that ".partial" extension. > > The 028_pitr_timelines tests are highlighting that fact: with test data being being in 000000020000000000000003 and 000000010000000000000003.partial, a recovery following the latest timeline (2) will succeed but fail if we follow the current timeline (1). > > By simply trying to fetch the ".partial" file in XLogFileRead, we can easily recover more data and also cover that (current timeline) recovery case. > > So, this proposed patch makes XLogFileRead try to restore ".partial" WAL archives and adds a test to 028_pitr_timelines using current recovery_target_timeline. Does this path only get hit when we don't already have any WAL segments (or partial segments) left for that timeline? I'm a bit worried about overwriting existing (partial) segments that may have more WAL than what we can get from archives. (patch v2) > + restoredArchivedFile = !RestoreArchivedFile(path, xlogfname, > + "RECOVERYXLOG", > + wal_segment_size, > + InRedo) && > + !RestoreArchivedFile(path, partialxlogfname, > "RECOVERYXLOG", > wal_segment_size, > - InRedo)) > + InRedo); The value of restoredArchiveFile is inverted with what it indicates: It is true when we failed to restore an archived xlog segment, and false if we did succeed. I'm also not a fan of the additional allocation of partialxlogfname in this code. It could well do without, by "just" reusing the xlogfname scratch space when we fail to recover the full segment. Kind regards, Matthias van de Meent Neon (https://neon.tech) ^ permalink raw reply [nested|flat] 4+ messages in thread
* [PATCH v22 8/8] Allow to print raw parse tree. @ 2024-09-19 04:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw) --- src/backend/tcop/postgres.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e394f1419a..1612d9fcca 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -659,6 +659,10 @@ pg_parse_query(const char *query_string) #endif /* DEBUG_NODE_TESTS_ENABLED */ + if (Debug_print_parse) + elog_node_display(LOG, "raw parse tree", raw_parsetree_list, + Debug_pretty_print); + TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string); return raw_parsetree_list; -- 2.25.1 ----Next_Part(Thu_Sep_19_13_59_47_2024_608)---- ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2024-10-18 13:56 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-04-05 09:45 Recovery of .partial WAL segments Stefan Fercot <[email protected]> 2024-08-02 06:47 ` Stefan Fercot <[email protected]> 2024-10-18 13:56 ` Matthias van de Meent <[email protected]> 2024-09-19 04:48 [PATCH v22 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox