Subject: Re: pg_waldump: support decoding of WAL inside tarfile

From: Amul Sul <[email-masked]>
Date: Tue, 10 Feb 2026 15:06:12 +0530

On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <[email-masked]> wrote:
>
> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <[email-masked]> wrote:
> >
> > On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <[email-masked]> wrote:
> > > In the attached version, I am using the WAL segment name as the hash
> > > key, which is much more straightforward. I have rewritten
> > > read_archive_wal_page(), and it looks much cleaner than before. The
> > > logic to discard irrelevant WAL files is still within
> > > get_archive_wal_entry. I added an explanation for setting cur_wal to
> > > NULL, which is now handled in the separate function I mentioned
> > > previously.
> > >
> > > Kindly have a look at the attached version; let me know if you are
> > > still not happy with the current approach for filtering/discarding
> > > irrelevant WAL segments. It isn't much different from the previous
> > > version, but I have tried to keep it in a separate routine for better
> > > code readability, with comments to make it easier to understand. I
> > > also added a comment for ArchivedWALFile.
> >
> > I feel like the division of labor between get_archive_wal_entry() and
> > read_archive_wal_page() is odd. I noticed this in the last version,
> > too, and it still seems to be the case. get_archive_wal_entry() first
> > calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> > If it doesn't, it loops until an entry for the requested file shows up
> > and then returns it. Then control returns to read_archive_wal_page()
> > which loops some more until we have all the data we need for the
> > requested file. But it seems odd to me to have two separate loops
> > here. I think that the first loop is going to call read_archive_file()
> > until we find the beginning of the file that we care about and then
> > the second one is going to call read_archive_file() some more until we
> > have read enough of it to satisfy the request. It feels odd to me to
> > do it that way, as if we told somebody to first wait until 9 o'clock
> > and then wait another 30 minutes, instead of just telling them to wait
> > until 9:30. I realize it's not quite the same thing, because apart
> > from calling read_archive_file(), the two loops do different things,
> > but I still think it looks odd.
> >
> > + /*
> > + * Ignore if the timeline is different or the current segment is not
> > + * the desired one.
> > + */
> > + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> > + if (privateInfo->timeline != curSegTimeline ||
> > + privateInfo->startSegNo > curSegNo ||
> > + privateInfo->endSegNo < curSegNo ||
> > + segno > curSegNo)
> > + {
> > + free_archive_wal_entry(entry->fname, privateInfo);
> > + continue;
> > + }
> >
> > The comment doesn't match the code. If it did, the test would be
> > (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> > instead the segno test is > rather than !=, and the checks against
> > startSegNo and endSegNo aren't explained at all. I think I understand
> > why the segno test uses > rather than !=, but it's the point of the
> > comment to explain things like that, rather than leaving the reader to
> > guess. And I don't know why we also need to test startSegNo and
> > endSegNo.
> >
> > I also wonder what the point is of doing XLogFromFileName() on the
> > fname provided by the caller and then again on entry->fname. Couldn't
> > you just compare the strings?
> >
> > Again, the division of labor is really odd here. It's the job of
> > astreamer_waldump_content() to skip things that aren't WAL files at
> > all, but it's the job of get_archive_wal_entry() to skip things that
> > are WAL files but not the one we want. I disagree with putting those
> > checks in completely separate parts of the code.
> >
>
> Keeping the timeline and segment start-end range checks inside the
> archive streamer creates a circular dependency that cannot be resolved
> without a 'dirty hack'. We must read the first available WAL file page
> to determine the wal_segment_size before it can calculate the target
> segment range. Moving the checks inside the streamer would make it
> impossible to process that initial file, as the necessary filtering
> parameters --  would still be unknown which would need to be skipped
> for the first read somehow. What if later we realized that the first
> WAL file which was allowed to be streamed by skipping that check is
> irrelevant and doesn't fall under the start-end segment range?
>

Please have a look at the attached version, specifically patch 0005.
In astreamer_waldump_content(), I have moved the WAL file filtration
check from get_archive_wal_entry(). This check will be skipped during
the initial read in init_archive_reader(), which instead performs it
explicitly once it determines the WAL segment size and the start/end
segments.

To access the WAL segment size inside astreamer_waldump_content(), I
have moved the WAL segment size variable into the XLogDumpPrivate
structure in the separate 0004 patch.

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Wed, 18 Feb 2026 12:28:27 +0530

> structure in the separate 0004 patch.

Attached is an updated version including the aforesaid changes. It
includes a new refactoring patch (0001) that moves the logic for
identifying tar archives and their compression types from
pg_basebackup and pg_verifybackup into a separate-reusable function,
per a suggestion from Euler [1].  Additionally, I have added a test
for the contrecord decoding to the main patch (now 0006).

1] http://postgr.es/m/[email-masked]

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Mon, 2 Mar 2026 18:30:24 +0530

> 1] http://postgr.es/m/[email-masked]

Rebased against the latest master, fixed typos in code comments, and
replaced palloc0 with palloc0_object.

Regards,
Amul

---

From: Andrew Dunstan <[email-masked]>
Date: Tue, 3 Mar 2026 19:37:48 -0500

> replaced palloc0 with palloc0_object.

Hi Amul.

I think this looks in pretty good shape.

Attached are patches for a few things I think could be fixed. They are 
mostly self-explanatory. The TAP test fix is the only sane way I could 
come up with stopping the skip code you had from reporting a wildly 
inaccurate number of tests skipped. The sane way to do this from a 
Test::More perspective is a subtest, but unfortunately meson does not 
like subtest output, which is why we don't use it elsewhere, so the only 
way I could come up with was to split this out into a separate test. Of 
course, we might just say we don't care about the misreport, in which 
case we could just live with things as they are.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Tue, 3 Mar 2026 00:00:00 +0000
Subject: [PATCH] Add pg_verifybackup test for tar-format WAL verification

The new tar-format WAL verification in pg_verifybackup had no test
coverage for the case where pg_basebackup produces a separate
pg_wal.tar (--format=tar --wal-method=stream).  Add a test that takes
a tar-format backup and verifies it.
---
 src/bin/pg_verifybackup/t/007_wal.pl | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/bin/pg_verifybackup/t/007_wal.pl b/src/bin/pg_verifybackup/t/007_wal.pl
index 8ad2234453d..0e0377bfacc 100644
--- a/src/bin/pg_verifybackup/t/007_wal.pl
+++ b/src/bin/pg_verifybackup/t/007_wal.pl
@@ -90,4 +90,20 @@ command_ok(
 	[ 'pg_verifybackup', $backup_path2 ],
 	'valid base backup with timeline > 1');
 
+# Test WAL verification for a tar-format backup with a separate pg_wal.tar,
+# as produced by pg_basebackup --format=tar --wal-method=stream.
+my $backup_path3 = $primary->backup_dir . '/test_tar_wal';
+$primary->command_ok(
+	[
+		'pg_basebackup',
+		'--pgdata' => $backup_path3,
+		'--no-sync',
+		'--format' => 'tar',
+		'--checkpoint' => 'fast'
+	],
+	"tar backup with separate pg_wal.tar");
+command_ok(
+	[ 'pg_verifybackup', $backup_path3 ],
+	'WAL verification succeeds with separate pg_wal.tar');
+
 done_testing();
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Tue, 3 Mar 2026 00:00:00 +0000
Subject: [PATCH] Split pg_waldump TAP tests into directory and archive files

The original 001_basic.pl mixed directory and tar archive tests in a
single SKIP loop with a hardcoded skip count of 3, but each scenario
actually runs ~19 assertions.  When tar is unavailable the skip count
was wrong, and the directory scenario was also wrongly guarded by the
tar-availability check.

Move all archive-related tests (tar, tar.gz) into a new
003_archive.pl that uses plan skip_all when tar is unavailable,
cleanly skipping the entire file.  001_basic.pl retains only
directory-based tests with no SKIP blocks needed.
---
 src/bin/pg_waldump/meson.build       |   1 +
 src/bin/pg_waldump/t/001_basic.pl    | 221 ++++++++++-----------------
 src/bin/pg_waldump/t/003_archive.pl  | 320 +++++++++++++++++++++++++++++++++++
 3 files changed, 396 insertions(+), 146 deletions(-)
 create mode 100644 src/bin/pg_waldump/t/003_archive.pl

diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..d2b4bd0c048 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -34,6 +34,7 @@ tests += {
     'tests': [
       't/001_basic.pl',
       't/002_save_fullpage.pl',
+      't/003_archive.pl',
     ],
   },
 }
diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 9854c939007..282c9a37221 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -3,13 +3,9 @@
 
 use strict;
 use warnings FATAL => 'all';
-use Cwd;
 use PostgreSQL::Test::Cluster;
 use PostgreSQL::Test::Utils;
 use Test::More;
-use List::Util qw(shuffle);
-
-my $tar = $ENV{TAR};
 
 program_help_ok('pg_waldump');
 program_version_ok('pg_waldump');
@@ -195,8 +191,8 @@ END
 $$;
 });
 
-my $contrecord_lsn = $node->safe_psql('postgres',
-	'SELECT pg_current_wal_insert_lsn()');
+my $contrecord_lsn =
+  $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
 # Generate contrecord record
 $node->safe_psql('postgres',
 	qq{SELECT pg_logical_emit_message(true, 'test 026', repeat('xyzxz', 123456))}
@@ -299,145 +295,78 @@ sub test_pg_waldump
 	return @lines;
 }
 
-# Create a tar archive, sorting the file order
-sub generate_archive
-{
-	my ($archive, $directory, $compression_flags) = @_;
-
-	my @files;
-	opendir my $dh, $directory or die "opendir: $!";
-	while (my $entry = readdir $dh) {
-		# Skip '.' and '..'
-		next if $entry eq '.' || $entry eq '..';
-		push @files, $entry;
-	}
-	closedir $dh;
-
-	@files = shuffle @files;
-
-	# move into the WAL directory before archiving files
-	my $cwd = getcwd;
-	chdir($directory) || die "chdir: $!";
-	command_ok([$tar, $compression_flags, $archive, @files]);
-	chdir($cwd) || die "chdir: $!";
-}
-
-my $tmp_dir = PostgreSQL::Test::Utils::tempdir_short();
-
-my @scenarios = (
-	{
-		'path' => $node->data_dir,
-		'is_archive' => 0,
-		'enabled' => 1
-	},
-	{
-		'path' => "$tmp_dir/pg_wal.tar",
-		'compression_method' => 'none',
-		'compression_flags' => '-cf',
-		'is_archive' => 1,
-		'enabled' => 1
-	},
-	{
-		'path' => "$tmp_dir/pg_wal.tar.gz",
-		'compression_method' => 'gzip',
-		'compression_flags' => '-czf',
-		'is_archive' => 1,
-		'enabled' => check_pg_config("#define HAVE_LIBZ 1")
-	});
-
-for my $scenario (@scenarios)
-{
-	my $path = $scenario->{'path'};
-
-	SKIP:
-	{
-		skip "tar command is not available", 3
-		  if !defined $tar;
-		skip "$scenario->{'compression_method'} compression not supported by this build", 3
-		  if !$scenario->{'enabled'} && $scenario->{'is_archive'};
-
-		  # create pg_wal archive
-		  if ($scenario->{'is_archive'})
-		  {
-			  generate_archive($path,
-				  $node->data_dir . '/pg_wal',
-				  $scenario->{'compression_flags'});
-		  }
-
-		command_fails_like(
-			[ 'pg_waldump', '--path' => $path ],
-			qr/error: no start WAL location given/,
-			'path option requires start location');
-		command_like(
-			[
-				'pg_waldump',
-				'--path' => $path,
-				'--start' => $start_lsn,
-				'--end' => $end_lsn,
-			],
-			qr/./,
-			'runs with path option and start and end locations');
-		command_fails_like(
-			[
-				'pg_waldump',
-				'--path' => $path,
-				'--start' => $start_lsn,
-			],
-			qr/error: error in WAL record at/,
-			'falling off the end of the WAL results in an error');
-
-		command_fails_like(
-			[
-				'pg_waldump', '--quiet',
-				'--path' => $path,
-				'--start' => $start_lsn
-			],
-			qr/error: error in WAL record at/,
-			'errors are shown with --quiet');
-
-		test_pg_waldump_skip_bytes($path, $start_lsn, $end_lsn);
-
-		my @lines = test_pg_waldump($path, $start_lsn, $end_lsn);
-		is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
-
-		@lines = test_pg_waldump($path, $contrecord_lsn, $end_lsn);
-		is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
-
-		test_pg_waldump_skip_bytes($path, $contrecord_lsn, $end_lsn);
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--limit' => 6);
-		is(@lines, 6, 'limit option observed');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fullpage');
-		is(grep(!/^rmgr:.*\bFPW\b/, @lines), 0, 'all output lines are FPW');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats');
-		like($lines[0], qr/WAL statistics/, "statistics on stdout");
-		is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats=record');
-		like($lines[0], qr/WAL statistics/, "statistics on stdout");
-		is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--rmgr' => 'Btree');
-		is(grep(!/^rmgr: Btree/, @lines), 0, 'only Btree lines');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fork' => 'init');
-		is(grep(!/fork init/, @lines), 0, 'only init fork lines');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn,
-			'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_t1_oid");
-		is(grep(!/rel $default_ts_oid\/$postgres_db_oid\/$rel_t1_oid/, @lines),
-			0, 'only lines for selected relation');
-
-		@lines = test_pg_waldump($path, $start_lsn, $end_lsn,
-			'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_i1a_oid",
-			'--block' => 1);
-		is(grep(!/\bblk 1\b/, @lines), 0, 'only lines for selected block');
-
-		# Cleanup.
-		unlink $path if $scenario->{'is_archive'};
-	}
-}
+my $path = $node->data_dir;
+
+command_fails_like(
+	[ 'pg_waldump', '--path' => $path ],
+	qr/error: no start WAL location given/,
+	'path option requires start location');
+command_like(
+	[
+		'pg_waldump',
+		'--path' => $path,
+		'--start' => $start_lsn,
+		'--end' => $end_lsn,
+	],
+	qr/./,
+	'runs with path option and start and end locations');
+command_fails_like(
+	[
+		'pg_waldump',
+		'--path' => $path,
+		'--start' => $start_lsn,
+	],
+	qr/error: error in WAL record at/,
+	'falling off the end of the WAL results in an error');
+
+command_fails_like(
+	[
+		'pg_waldump', '--quiet',
+		'--path' => $path,
+		'--start' => $start_lsn
+	],
+	qr/error: error in WAL record at/,
+	'errors are shown with --quiet');
+
+test_pg_waldump_skip_bytes($path, $start_lsn, $end_lsn);
+
+my @lines = test_pg_waldump($path, $start_lsn, $end_lsn);
+is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
+
+@lines = test_pg_waldump($path, $contrecord_lsn, $end_lsn);
+is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
+
+test_pg_waldump_skip_bytes($path, $contrecord_lsn, $end_lsn);
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--limit' => 6);
+is(@lines, 6, 'limit option observed');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fullpage');
+is(grep(!/^rmgr:.*\bFPW\b/, @lines), 0, 'all output lines are FPW');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats');
+like($lines[0], qr/WAL statistics/, "statistics on stdout");
+is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats=record');
+like($lines[0], qr/WAL statistics/, "statistics on stdout");
+is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--rmgr' => 'Btree');
+is(grep(!/^rmgr: Btree/, @lines), 0, 'only Btree lines');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fork' => 'init');
+is(grep(!/fork init/, @lines), 0, 'only init fork lines');
+
+@lines = test_pg_waldump($path, $start_lsn, $end_lsn,
+	'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_t1_oid");
+is(grep(!/rel $default_ts_oid\/$postgres_db_oid\/$rel_t1_oid/, @lines),
+	0, 'only lines for selected relation');
+
+@lines = test_pg_waldump(
+	$path, $start_lsn, $end_lsn,
+	'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_i1a_oid",
+	'--block' => 1);
+is(grep(!/\bblk 1\b/, @lines), 0, 'only lines for selected block');
 
 done_testing();
new file mode 100644
index 00000000000..c615713efd4
--- /dev/null
+++ b/src/bin/pg_waldump/t/003_archive.pl
@@ -0,0 +1,320 @@
+
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+# Test pg_waldump's ability to read WAL from tar archives.
+
+use strict;
+use warnings FATAL => 'all';
+use Cwd;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use List::Util qw(shuffle);
+
+my $tar = $ENV{TAR};
+
+if (!defined $tar)
+{
+	plan skip_all => 'tar command is not available';
+}
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+	'postgresql.conf', q{
+autovacuum = off
+checkpoint_timeout = 1h
+
+# for standbydesc
+archive_mode=on
+archive_command=''
+
+# for XLOG_HEAP_TRUNCATE
+wal_level=logical
+});
+$node->start;
+
+my ($start_lsn, $start_walfile) = split /\|/,
+  $node->safe_psql('postgres',
+	q{SELECT pg_current_wal_insert_lsn(), pg_walfile_name(pg_current_wal_insert_lsn())}
+  );
+
+$node->safe_psql(
+	'postgres', q{
+-- heap, btree, hash, sequence
+CREATE TABLE t1 (a int GENERATED ALWAYS AS IDENTITY, b text);
+CREATE INDEX i1a ON t1 USING btree (a);
+CREATE INDEX i1b ON t1 USING hash (b);
+INSERT INTO t1 VALUES (default, 'one'), (default, 'two');
+DELETE FROM t1 WHERE b = 'one';
+TRUNCATE t1;
+
+-- abort
+START TRANSACTION;
+INSERT INTO t1 VALUES (default, 'three');
+ROLLBACK;
+
+-- unlogged/init fork
+CREATE UNLOGGED TABLE t2 (x int);
+CREATE INDEX i2 ON t2 USING btree (x);
+INSERT INTO t2 SELECT generate_series(1, 10);
+
+-- gin
+CREATE TABLE gin_idx_tbl (id bigserial PRIMARY KEY, data jsonb);
+CREATE INDEX gin_idx ON gin_idx_tbl USING gin (data);
+INSERT INTO gin_idx_tbl
+    WITH random_json AS (
+        SELECT json_object_agg(key, trunc(random() * 10)) as json_data
+            FROM unnest(array['a', 'b', 'c']) as u(key))
+          SELECT generate_series(1,500), json_data FROM random_json;
+
+-- gist, spgist
+CREATE TABLE gist_idx_tbl (p point);
+CREATE INDEX gist_idx ON gist_idx_tbl USING gist (p);
+CREATE INDEX spgist_idx ON gist_idx_tbl USING spgist (p);
+INSERT INTO gist_idx_tbl (p) VALUES (point '(1, 1)'), (point '(3, 2)'), (point '(6, 3)');
+
+-- brin
+CREATE TABLE brin_idx_tbl (col1 int, col2 text, col3 text );
+CREATE INDEX brin_idx ON brin_idx_tbl USING brin (col1, col2, col3) WITH (autosummarize=on);
+INSERT INTO brin_idx_tbl SELECT generate_series(1, 10000), 'dummy', 'dummy';
+UPDATE brin_idx_tbl SET col2 = 'updated' WHERE col1 BETWEEN 1 AND 5000;
+SELECT brin_summarize_range('brin_idx', 0);
+SELECT brin_desummarize_range('brin_idx', 0);
+
+VACUUM;
+
+-- logical message
+SELECT pg_logical_emit_message(true, 'foo', 'bar');
+
+-- relmap
+VACUUM FULL pg_authid;
+
+-- database
+CREATE DATABASE d1;
+DROP DATABASE d1;
+});
+
+my $tblspc_path = PostgreSQL::Test::Utils::tempdir_short();
+
+$node->safe_psql(
+	'postgres', qq{
+CREATE TABLESPACE ts1 LOCATION '$tblspc_path';
+DROP TABLESPACE ts1;
+});
+
+# Consume all remaining room in the current WAL segment, leaving space enough
+# only for the start of a largish record, to test contrecord decoding.
+$node->safe_psql(
+	'postgres', q{
+DO $$
+DECLARE
+    wal_segsize int := setting::int FROM pg_settings WHERE name = 'wal_segment_size';
+    remain int;
+    iters  int := 0;
+BEGIN
+    LOOP
+        INSERT into t1(b)
+        select repeat(encode(sha256(g::text::bytea), 'hex'), (random() * 15 + 1)::int)
+        from generate_series(1, 10) g;
+
+        remain := wal_segsize - (pg_current_wal_insert_lsn() - '0/0') % wal_segsize;
+        IF remain < 2 * setting::int from pg_settings where name = 'block_size' THEN
+            RAISE log 'exiting after % iterations, % bytes to end of WAL segment', iters, remain;
+            EXIT;
+        END IF;
+        iters := iters + 1;
+    END LOOP;
+END
+$$;
+});
+
+my $contrecord_lsn =
+  $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
+$node->safe_psql('postgres',
+	qq{SELECT pg_logical_emit_message(true, 'test 026', repeat('xyzxz', 123456))}
+);
+
+my ($end_lsn, $end_walfile) = split /\|/,
+  $node->safe_psql('postgres',
+	q{SELECT pg_current_wal_insert_lsn(), pg_walfile_name(pg_current_wal_insert_lsn())}
+  );
+
+$node->stop;
+
+
+sub test_pg_waldump_skip_bytes
+{
+	my ($path, $startlsn, $endlsn) = @_;
+
+	my ($part1, $part2) = split qr{/}, $startlsn;
+	my $lsn2 = hex $part2;
+	$lsn2++;
+	my $new_start = sprintf("%s/%X", $part1, $lsn2);
+
+	my ($stdout, $stderr);
+
+	my $result = IPC::Run::run [
+		'pg_waldump',
+		'--start' => $new_start,
+		'--end' => $endlsn,
+		'--path' => $path,
+	  ],
+	  '>' => \$stdout,
+	  '2>' => \$stderr;
+	ok($result, "runs with start segment and start LSN specified");
+	like($stderr, qr/first record is after/, 'info message printed');
+}
+
+sub test_pg_waldump
+{
+	local $Test::Builder::Level = $Test::Builder::Level + 1;
+	my ($path, $startlsn, $endlsn, @opts) = @_;
+
+	my ($stdout, $stderr);
+
+	my $result = IPC::Run::run [
+		'pg_waldump',
+		'--start' => $startlsn,
+		'--end' => $endlsn,
+		'--path' => $path,
+		@opts
+	  ],
+	  '>' => \$stdout,
+	  '2>' => \$stderr;
+	ok($result, "pg_waldump @opts: runs ok");
+	is($stderr, '', "pg_waldump @opts: no stderr");
+	my @lines = split /\n/, $stdout;
+	ok(@lines > 0, "pg_waldump @opts: some lines are output");
+	return @lines;
+}
+
+sub generate_archive
+{
+	my ($archive, $directory, $compression_flags) = @_;
+
+	my @files;
+	opendir my $dh, $directory or die "opendir: $!";
+	while (my $entry = readdir $dh)
+	{
+		next if $entry eq '.' || $entry eq '..';
+		push @files, $entry;
+	}
+	closedir $dh;
+
+	@files = shuffle @files;
+
+	my $cwd = getcwd;
+	chdir($directory) || die "chdir: $!";
+	command_ok([ $tar, $compression_flags, $archive, @files ],
+		"create archive $archive");
+	chdir($cwd) || die "chdir: $!";
+}
+
+
+my $tmp_dir = PostgreSQL::Test::Utils::tempdir_short();
+
+my @scenarios = (
+	{
+		'path' => "$tmp_dir/pg_wal.tar",
+		'compression_method' => 'none',
+		'compression_flags' => '-cf',
+		'enabled' => 1,
+	},
+	{
+		'path' => "$tmp_dir/pg_wal.tar.gz",
+		'compression_method' => 'gzip',
+		'compression_flags' => '-czf',
+		'enabled' => check_pg_config("#define HAVE_LIBZ 1"),
+	});
+
+for my $scenario (@scenarios)
+{
+	my $path = $scenario->{'path'};
+	my $method = $scenario->{'compression_method'};
+
+  SKIP:
+	{
+		skip "$method compression not supported by this build", 1
+		  if !$scenario->{'enabled'};
+
+		generate_archive(
+			$path,
+			$node->data_dir . '/pg_wal',
+			$scenario->{'compression_flags'});
+
+		command_fails_like(
+			[ 'pg_waldump', '--path' => $path ],
+			qr/error: no start WAL location given/,
+			"$method: path option requires start location");
+		command_like(
+			[
+				'pg_waldump',
+				'--path' => $path,
+				'--start' => $start_lsn,
+				'--end' => $end_lsn,
+			],
+			qr/./,
+			"$method: runs with path option and start and end locations");
+		command_fails_like(
+			[
+				'pg_waldump',
+				'--path' => $path,
+				'--start' => $start_lsn,
+			],
+			qr/error: error in WAL record at/,
+			"$method: falling off the end of the WAL results in an error");
+
+		command_fails_like(
+			[
+				'pg_waldump', '--quiet',
+				'--path' => $path,
+				'--start' => $start_lsn
+			],
+			qr/error: error in WAL record at/,
+			"$method: errors are shown with --quiet");
+
+		test_pg_waldump_skip_bytes($path, $start_lsn, $end_lsn);
+
+		my @lines = test_pg_waldump($path, $start_lsn, $end_lsn);
+		is(grep(!/^rmgr: \w/, @lines),
+			0, "$method: all output lines are rmgr lines");
+
+		@lines = test_pg_waldump($path, $contrecord_lsn, $end_lsn);
+		is(grep(!/^rmgr: \w/, @lines),
+			0, "$method: contrecord - all output lines are rmgr lines");
+
+		test_pg_waldump_skip_bytes($path, $contrecord_lsn, $end_lsn);
+
+		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--limit' => 6);
+		is(@lines, 6, "$method: limit option observed");
+
+		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fullpage');
+		is(grep(!/^rmgr:.*\bFPW\b/, @lines),
+			0, "$method: all output lines are FPW");
+
+		@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats');
+		like($lines[0], qr/WAL statistics/, "$method: statistics on stdout");
+		is(grep(/^rmgr:/, @lines), 0, "$method: no rmgr lines output");
+
+		@lines =
+		  test_pg_waldump($path, $start_lsn, $end_lsn, '--stats=record');
+		like($lines[0], qr/WAL statistics/,
+			"$method: stats=record on stdout");
+		is(grep(/^rmgr:/, @lines),
+			0, "$method: no rmgr lines with stats=record");
+
+		@lines =
+		  test_pg_waldump($path, $start_lsn, $end_lsn, '--rmgr' => 'Btree');
+		is(grep(!/^rmgr: Btree/, @lines), 0, "$method: only Btree lines");
+
+		@lines =
+		  test_pg_waldump($path, $start_lsn, $end_lsn, '--fork' => 'init');
+		is(grep(!/fork init/, @lines), 0, "$method: only init fork lines");
+
+		# Cleanup.
+		unlink $path;
+	}
+}
+
+done_testing();
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Tue, 3 Mar 2026 00:00:00 +0000
Subject: [PATCH] Fix documentation for pg_waldump tar archive support

Two documentation issues with the tar archive reading feature:

- pg_waldump.sgml: When reading WAL from a tar archive with
  out-of-order segments, pg_waldump spills to temporary files.  TMPDIR
  controls where those files are created, but this was not documented
  in the Environment section.

- pg_verifybackup.sgml: The --wal-path option description still only
  said "directory" even though it now also accepts tar archives.
---
 doc/src/sgml/ref/pg_verifybackup.sgml |  7 ++++---
 doc/src/sgml/ref/pg_waldump.sgml      | 11 +++++++++++
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/ref/pg_verifybackup.sgml b/doc/src/sgml/ref/pg_verifybackup.sgml
index 16b50b5a4df..1695cfe91c8 100644
--- a/doc/src/sgml/ref/pg_verifybackup.sgml
+++ b/doc/src/sgml/ref/pg_verifybackup.sgml
@@ -261,9 +261,10 @@ PostgreSQL documentation
       <term><option>--wal-path=<replaceable class="parameter">path</replaceable></option></term>
       <listitem>
        <para>
-        Try to parse WAL files stored in the specified directory, rather than
-        in <literal>pg_wal</literal>. This may be useful if the backup is
-        stored in a separate location from the WAL archive.
+        Try to parse WAL files stored in the specified directory or tar
+        archive, rather than in <literal>pg_wal</literal>. This may be
+        useful if the backup is stored in a separate location from the WAL
+        archive.
        </para>
       </listitem>
      </varlistentry>
diff --git a/doc/src/sgml/ref/pg_waldump.sgml b/doc/src/sgml/ref/pg_waldump.sgml
index b36323dde92..9bbb4bd5772 100644
--- a/doc/src/sgml/ref/pg_waldump.sgml
+++ b/doc/src/sgml/ref/pg_waldump.sgml
@@ -391,6 +391,17 @@ PostgreSQL documentation
      </para>
     </listitem>
    </varlistentry>
+
+   <varlistentry>
+    <term><envar>TMPDIR</envar></term>
+    <listitem>
+     <para>
+      Directory in which to create temporary files when reading WAL from a
+      tar archive with out-of-order segment files. If not set, the temporary
+      directory is created within the same directory as the tar archive.
+     </para>
+    </listitem>
+   </varlistentry>
   </variablelist>
  </refsect1>
 
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Tue, 3 Mar 2026 00:00:00 +0000
Subject: [PATCH] Fix bugs in pg_waldump tar archive support

Fix several bugs introduced by the pg_waldump archive WAL reading
feature:

- pg_waldump.c: The error path for verify_directory() printed waldir
  (which is NULL when --path is used) instead of walpath.

- archive_waldump.c: The error message for short reads had an operator
  precedence bug: (long long int) count - nbytes cast only count, not
  the subtraction result.  Also reported nbytes (the requested amount)
  instead of count (the total file size) for the "of" portion.

- archive_waldump.c: The "ignoring duplicate WAL" code path leaked
  fname (allocated via pnstrdup/palloc).  Also changed the existing
  free(fname) to pfree(fname) for consistency.

- pg_verifybackup.c: The rename from --wal-directory to --wal-path
  didn't preserve the old spelling as a backward-compatible alias.

- pg_verifybackup.c: Fix double space before "Or" in --wal-path
  error hint message.
---
diff --git a/src/bin/pg_verifybackup/pg_verifybackup.c b/src/bin/pg_verifybackup/pg_verifybackup.c
index 935ab8fafa8..b0b764913cf 100644
--- a/src/bin/pg_verifybackup/pg_verifybackup.c
+++ b/src/bin/pg_verifybackup/pg_verifybackup.c
@@ -131,6 +131,7 @@ main(int argc, char **argv)
 		{"quiet", no_argument, NULL, 'q'},
 		{"skip-checksums", no_argument, NULL, 's'},
 		{"wal-path", required_argument, NULL, 'w'},
+		{"wal-directory", required_argument, NULL, 'w'},
 		{NULL, 0, NULL, 0}
 	};
 
@@ -376,7 +377,7 @@ main(int argc, char **argv)
 		else
 		{
 			pg_log_error("WAL archive not found");
-			pg_log_error_hint("Specify the correct path using the option -w/--wal-path.  "
+			pg_log_error_hint("Specify the correct path using the option -w/--wal-path. "
 							  "Or you must use -n/--no-parse-wal when verifying a tar-format backup.");
 			exit(1);
 		}
diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index c5a4485b5b1..1479efe61f5 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -344,8 +344,8 @@ read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
 				read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
 				pg_fatal("could not read file \"%s\" from archive \"%s\": read %lld of %lld",
 						 fname, privateInfo->archive_name,
-						 (long long int) count - nbytes,
-						 (long long int) nbytes);
+						 (long long int) (count - nbytes),
+						 (long long int) count);
 		}
 	}
 
@@ -664,7 +664,7 @@ astreamer_waldump_content(astreamer *streamer, astreamer_member *member,
 						privateInfo->start_segno > segno ||
 						privateInfo->end_segno < segno)
 					{
-						free(fname);
+						pfree(fname);
 						break;
 					}
 				}
@@ -680,6 +680,7 @@ astreamer_waldump_content(astreamer *streamer, astreamer_member *member,
 				{
 					pg_log_warning("ignoring duplicate WAL \"%s\" found in archive \"%s\"",
 								   member->pathname, privateInfo->archive_name);
+					pfree(fname);
 					break;
 				}
 
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 114969217d8..4b438b53ead 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -1223,7 +1223,7 @@ main(int argc, char **argv)
 		/* validate path points to directory */
 		else if (!verify_directory(walpath))
 		{
-			pg_log_error("could not open directory \"%s\": %m", waldir);
+			pg_log_error("could not open directory \"%s\": %m", walpath);
 			goto bad_argument;
 		}
 	}

---

From: Amul Sul <[email-masked]>
Date: Wed, 4 Mar 2026 18:22:00 +0530

> I think this looks in pretty good shape.

Thank you very much for looking at the patch.

> case we could just live with things as they are.

I agree that the reported skip number was incorrect, and I have
corrected it in the attached patch. I haven't applied your patch for
the TAP test improvements yet because I wanted to double-check it
first with you; the patch as it stood created duplicate tests already
present in 001_basic.pl. To avoid this duplication, I have added a
loop that performs tests for both plain and tar WAL directory inputs,
similar to the approach used in pg_verifybackup for different
compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
don't have any objection to doing so if you feel the duplication is
acceptable, but I feel that using a loop for the tests in 001_basic.pl
is a bit tidier. Let me know your thoughts.

I have applied all your other patches but skipped the changes to
pg_verifybackup.c from cf5955-fixes.patch.no-cfbot, as they seem
unrelated or perhaps I have misunderstood them.

Regards,
Amul

---

From: Andrew Dunstan <[email-masked]>
Date: Wed, 4 Mar 2026 16:50:30 -0500

On 2026-03-04 We 7:52 AM, Amul Sul wrote:
> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<[email-masked]> wrote:
>>
>>> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<[email-masked]> wrote:
>>>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<[email-masked]> wrote:
>>>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<[email-masked]> wrote:
>>>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<[email-masked]> wrote:
>>>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<[email-masked]> wrote:
>>>> 1]http://postgr.es/m/[email-masked]
>>>>

I will take a look.

> unrelated or perhaps I have misunderstood them.

<brown-paper-bag> That's what I get for using a poorly written tool.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

---

From: Amul Sul <[email-masked]>
Date: Mon, 9 Mar 2026 17:56:50 +0530

On Sat, Mar 7, 2026 at 3:51 AM Andrew Dunstan <[email-masked]> wrote:
>
>
> On 2026-03-04 We 4:50 PM, Andrew Dunstan wrote:
> >
> >
> I'm ok, with doing it this way. It's just a bit fragile - if we add a
> test the number will be wrong. But maybe it's not worth worrying about.
>
> Everything else looks fairly good. The attached fixes a few relatively
> minor issues in v15. The main one is that it stops allocating/freeing a
> buffer every time we call read_archive_file() and instead adds a
> reusable buffer. It also adds back wal-directory as an undocumented
> alias of wal-path, to avoid breaking legacy scripts unnecessarily, and
> adds constness to the fname argument of pg_tar_compress_algorithm, as
> well as fixing some indentation and grammar issues.
>
> All in all I think we're in good shape.

Thanks for the review. I have incorporated your suggested changes,
with one exception: I have skipped the buffer reallocation code in
read_archive_file(). Since we only handle two specific read sizes --
XLOG_BLCKSZ and READ_CHUNK_SIZE (128 KB, we defined in
archive_waldump.c) -- dynamic reallocation seems unnecessary. Instead,
I moved the allocation to init_archive_reader(), which now initializes
a buffer at READ_CHUNK_SIZE. I also added an assertion in
read_archive_file() to ensure that no read request exceeds this
allocated capacity.

Kindly have a look at the attached version and let me know your thoughts.

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Wed, 18 Mar 2026 17:15:47 +0530

On Wed, Mar 11, 2026 at 10:38 PM Andrew Dunstan <[email-masked]> wrote:
>
>
> On 2026-03-09 Mo 8:26 AM, Amul Sul wrote:
>
> Looks pretty good. I have squashed them into three patches I think are committable. Also attached is a diff showing what's changed - mainly this:
>
> . --follow + tar archive rejected (pg_waldump.c) — new validation prevents a confusing pg_fatal when combining --follow with a tar archive
> . error messages split (archive_waldump.c) — the single "could not read file" error is now two distinct messages: "WAL segment is too short" (truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an issue raised in review
> . hash table cleanup (archive_waldump.c) — free_archive_reader now iterates and frees all remaining hash entries and destroys the table
>

The final squashed version looks good to me, thank you. But, I would
like to propose splitting the 0001 patch into two separate commits: a
preparatory refactoring of the pg_waldump code and a standalone commit
that moves the tar archive detection and compression logic to a common
location, as the latter is an independent improvement to the existing
codebase. Additionally, since the test file refactoring was only kept
separate to facilitate the review and has already been reviewed, I
suggest merging those changes into the main feature patch i.e. 0002.
All other elements should remain in a single preparatory refactoring
patch for pg_waldump.

Attached is the version that includes the proposed split. No
additional changes to 0002 and 0003 patches.

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Wed, 18 Mar 2026 20:46:35 +0530

On Wed, Mar 18, 2026 at 5:15 PM Amul Sul <[email-masked]> wrote:
> > [...]

Added the two missing 'Reviewed-by' lines to the credit section of the
commit message and did a minor optimization in get_archive_wal_entry.

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Thu, 19 Mar 2026 15:50:59 +0530

> commit message and did a minor optimization in get_archive_wal_entry.

Attaching an updated version. It includes some tweaks to code
comments, adds an assert inside get_archive_wal_entry(), moves the
archive_read_buf_size declaration and usage into an assert-enabled
check, and makes a minor change to precheck_tar_backup_file() to
assign out-variables only after successful validation.

Regards,
Amul

---

From: Zsolt Parragi <[email-masked]>
Date: Thu, 19 Mar 2026 20:48:27 +0000

Hello!

Path is ignored with a positional argument, I think this is a bug?

This fails:

pg_waldump --path /wal/dir 000000010000000000000001

And this works:

pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8

+{
+ int fname_len = strlen(fname);
+

Shouldn't this use size_t?

+ /*
+ * Setup temporary directory to store WAL segments and set up an exit
+ * callback to remove it upon completion.
+ */
+ setup_tmpwal_dir(waldir);

Maybe this could be deferred to be created only on first use? If I
understand correctly, in a typical scenario waldump won't use this
temporary directory, yet it always creates it.

---

From: Amul Sul <[email-masked]>
Date: Fri, 20 Mar 2026 17:01:00 +0530

> pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8

Good catch! I've fixed this in the attached version and updated a test
case to cover this scenario.

> Shouldn't this use size_t?

Okay, that can be used. I’ve done the same in the attached version.

> temporary directory, yet it always creates it.

Yeah, that optimization can be done, but passing the waldir  -- which
is only used once -- to the point where the first temp file is created
would require quite a bit of code refactoring that doesn't seem to
offer much gain, IMO.

Regards,
Amul

---

From: Amul Sul <[email-masked]>
Date: Fri, 20 Mar 2026 18:56:42 +0530

> would require quite a bit of code refactoring that doesn't seem to

Since Andrew also leans toward creating the directory only when
needed, I have reconsidered the approach. I think we can pass waldir
(the archive directory) via XLogDumpPrivate, and I’ve implemented that
in the attached version.

Regards,
Amul

---

From: Andrew Dunstan <[email-masked]>
Date: Fri, 20 Mar 2026 15:33:18 -0400

> (the archive directory) via XLogDumpPrivate, and I’ve implemented that

Thanks, committed with very minor tweaks.

cheers

andrew

---

From: Amul Sul <[email-masked]>
Date: Sat, 21 Mar 2026 11:49:28 +0530

On Sat, Mar 21, 2026 at 9:19 AM Tom Lane <[email-masked]> wrote:
>
> Andrew Dunstan <[email-masked]> writes:
> Buildfarm members batta and hachi don't like this very much.
> They fail the pg_verifybackup tests like so:
>
> # Running: pg_verifybackup --exit-on-error /home/admin/batta/buildroot/HEAD/pgsql.build/src/bin/pg_verifybackup/tmp_check/t_008_untar_primary_data/backup/server-backup
> pg_waldump: error: could not find WAL in archive "base.tar.zst"
> pg_verifybackup: error: WAL parsing failed for timeline 1
>
> Only the zstd-compression case fails.  I've spent several hours trying
> to reproduce this, without any luck, although I can get a similar
> failure in only the gzip case if I build with --with-wal-blocksize=64.
> I do not have an explanation for the seeming cross-platform
> difference.  However after adding a lot of debug tracing, I believe
> I see the bug, or at least a related bug.  This bit in
> archive_waldump.c's init_archive_reader is where the error comes from:
>
>     /*
>      * Read until we have at least one full WAL page (XLOG_BLCKSZ bytes) from
>      * the first WAL segment in the archive so we can extract the WAL segment
>      * size from the long page header.
>      */
>     while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
>     {
>         if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
>             pg_fatal("could not find WAL in archive \"%s\"",
>                      privateInfo->archive_name);
>
>         entry = privateInfo->cur_file;
>     }
>
> That looks plausible but is in fact utterly broken when there's not a
> lot of WAL data in the archive, as there is not in this test case.
> There are at least two problems:
>

Thanks for the detailed debugging. I noticed the failure this morning
and had started investigating the issue, but in the meantime, I got
your helpful reply, which saved me a bunch of time and energy.

> 1. read_archive_file reads some data from the source WAL archive and
> shoves it into the astreamer decompression pipeline.  However, once it
> runs out of source data, it just returns zero and we fail immediately.
> This does not account for the possibility --- nay, certainty --- that
> there is data queued inside the decompression pipeline.  So this
> doesn't work if the data we need has been compressed into less than
> XLOG_BLCKSZ worth of compressed data.  (I suppose that the seeming
> cross-platform differences have to do with the effectiveness of the
> compression algorithm, but I don't really understand why it'd not be
> the same everywhere.)  We need to do astreamer_finalize once we run
> out of source data.  I think the cleanest place to handle that would
> be inside read_archive_file, but its return convention will need some
> rework if we want to put it there (because rc == 0 shouldn't cause an
> immediate failure if we were able to finalize some more data).  As an
> ugly experiment I put an astreamer_finalize call into the rc == 0 path
> of the above loop, but it still didn't work, because:
>
> 2. If the decompression pipeline reaches the end of the WAL file that
> we want, the ASTREAMER_MEMBER_TRAILER case in
> astreamer_waldump_content instantly resets privateInfo->cur_file to
> NULL.  Then the loop in init_archive_reader cannot exit successfully,
> and it will just read till the end of the archive and fail.
>
> I see that of the three callers of read_archive_file, only
> get_archive_wal_entry is aware of this possibility; but
> init_archive_reader certainly needs to deal with it and I bet
> read_archive_wal_page does too.  Moreover, get_archive_wal_entry's
> solution looks to me like a fragile kluge that probably doesn't work
> reliably either, the reason being that privateInfo->cur_file can
> change multiple times during a single call to read_archive_file,
> if the WAL data has been compressed sufficiently.  That whole API
> seems to need some rethinking, not to mention better documentation
> than the zero it has now.
>

I agree; init_archive_reader needs that handling, but
read_archive_wal_page doesn't need any fix. Since it only deals with
the current entry and already holds a reference to it, there is no
need to fetch it from the hash table again.

init_archive_reader has to scan the hash table because it doesn't
already have the specific WAL filename it is looking for, unlike
get_archive_wal_entry. Please have a look at the attached patch, which
tries to fix that.

> While I'm bitching: this error message "could not find WAL in archive
> \"%s\"" seems to me to be completely misleading and off-point.
>

I tried to improve that in the attached version.

regards,
Amul

---

From: Michael Paquier <[email-masked]>
Date: Sat, 21 Mar 2026 15:23:41 +0900

> pg_verifybackup: error: WAL parsing failed for timeline 1

I did not look at what's happening on the host, but it seems like a
safe bet to assume that we are not seeing many failures in the
buildfarm because we don't have many animals that have the idea to add
--with-zstd to their build configuration, like these two ones.
--
Michael

---

From: Amul Sul <[email-masked]>
Date: Sat, 21 Mar 2026 21:05:50 +0530

On Sat, Mar 21, 2026 at 5:51 PM Andrew Dunstan <[email-masked]> wrote:
>
>
> On 2026-03-21 Sa 2:34 AM, Tom Lane wrote:
>
> Michael Paquier <[email-masked]> writes:
>
> That may be part of the story, but only part.  I spent a good deal of
> time trying to reproduce batta & hachi's configurations locally, on
> several different platforms, but still couldn't duplicate what they
> are showing.
>
>
>
>
>
> Yeah, I haven't been able to reproduce it either. But while investigating I found a couple of issues. We neglected to add one of the tests to meson.build, and we neglected to close some files, causing errors on windows.
>

While the proposed fix of closing the file pointer before returning is
correct, we also need to ensure the file is reopened in the next call
to spill any remaining buffered data. I’ve made a small update to
Andrew's 0001 patch to handle this. Also, changes to meson.build don't
seem to be needed as we haven't committed that file yet (unless I am
missing something).

I’ve also reattached the other patches so they don't get lost: v2-0002
is Andrew's patch for the archive streamer, and v2-0003 is the patch I
posted previously [1].

Regards,
Amul

1] http://postgr.es/m/[email-masked]

---

From: Amul Sul <[email-masked]>
Date: Sat, 21 Mar 2026 22:56:38 +0530

> is Andrew's patch for the archive streamer, and v2-0003 is the patch I

On further thought, I don't think v2-0001 is the right patch. Consider
the case where we write a temporary file partially: if the next
segment required for decoding is that same segment,
TarWALDumpReadPage() will find the physical file present and continue
decoding, potentially triggering an error later due to the shorter
file.

I have attached the v3-0001 patch, which ensures that once we start
writing a temporary file, it should be finished before performing the
lookup. This ensures we don't leave a partial file on disk.

Updated patches are attached; 0002 and 0003 remain the same as before.

Regards,
Amul

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 07:24:56 -0400

On Sun, Mar 22, 2026 at 12:24 AM Tom Lane <[email-masked]> wrote:

> I wrote:
> > Unsurprisingly, applying this change to unmodified master results
> > in the pg_waldump and pg_verifybackup tests falling over.  More
> > surprisingly, they still fall over after applying your fix to the
> > decompressors, so there's some other source of garbage trailing
> > data.  I haven't figured out what.
>
> In the learn-something-new-every-day dept.: good ol' GNU tar itself
> does that.  By default, it zero-pads its output to a multiple of 10kB
> after it's written the required terminator.  Moreover, this behavior
> is actually specified by POSIX:
>
>   -x format
>     Specify the output archive format. The pax utility shall support
>     the following formats:
>     ...
>     ustar
>       The tar interchange format; see the EXTENDED DESCRIPTION
>       section. The default blocksize for this format for character
>       special archive files shall be 10240. Implementations shall
>       support all blocksize values less than or equal to 32256 that
>       are multiples of 512.
>
> So, astreamer_tar_parser_content's idea that it should disallow more
> than 1024 bytes of trailer is completely wrong, which we would have
> figured out long ago if the code attempting to enforce that weren't
> completely broken.
>
> You could argue that this means the tar files our existing utilities
> create aren't POSIX-compliant.  I think it's all right though: we
> can just say that we write these files with blocksize 1024 not
> blocksize 10240, and tar-file readers are required to accept that
> per the above spec text.
>
> However, this discourages me from editorializing on the file trailer
> emitted by whatever wrote the tar file we are reading.  I think
> emitting it as-is is the most appropriate thing.  So we should just
> get rid of astreamer_tar_parser_content's nonfunctional error check
> and not change its behavior otherwise.
>
>
>
OK, patch 5 of this set does that. I reworked your previous patches 2 and 3
slightly - mostly additional comments, and fixing a bug in use
of sizeof(XLogLongPageHeader). Patch 4 here tries to fix the wrong use of
cur_file in get_archive_wal_entry()

cheers

andrew
From 51d53b166df7c8eaebe49756e24088c16764807b Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 06:53:25 -0400
Subject: [PATCH v5 3/5] Fix init_archive_reader to not depend on cur_file.

init_archive_reader() relied on privateInfo->cur_file to track which
WAL segment was being read, but cur_file can become NULL if a member
trailer is processed during a read_archive_file() call.  This could
cause unreproducible "could not find WAL in archive" failures,
particularly with compressed archives where all the WAL data fits
in a small number of compressed bytes.

Fix by scanning the hash table after each read to find any cached
WAL segment with sufficient data, instead of depending on cur_file.
Also reduce the minimum data requirement from XLOG_BLCKSZ to
sizeof(XLogLongPageHeaderData), since we only need the long page
header to extract the segment size.

Add a safety comment on cur_file in pg_waldump.h to document that
it can change during a single read_archive_file() call.

Author: Tom Lane <[email-masked]>

Discussion: https://postgr.es/m/[email-masked]
---
 src/bin/pg_waldump/archive_waldump.c | 22 +++++++++++++++++-----
 src/bin/pg_waldump/pg_waldump.h      |  9 ++++++++-
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index cd092a057ef..3fce2183099 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -173,17 +173,29 @@ init_archive_reader(XLogDumpPrivate *privateInfo,
 	privateInfo->archive_wal_htab = ArchivedWAL_create(8, NULL);
 
 	/*
-	 * Read until we have at least one full WAL page (XLOG_BLCKSZ bytes) from
-	 * the first WAL segment in the archive so we can extract the WAL segment
-	 * size from the long page header.
+	 * Read until we have at least one WAL segment with enough data to extract
+	 * the WAL segment size from the long page header.
+	 *
+	 * We must not rely on cur_file here, because it can become NULL if a
+	 * member trailer is processed during a read_archive_file() call. Instead,
+	 * scan the hash table after each read to find any entry with sufficient
+	 * data.
 	 */
-	while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
+	while (entry == NULL)
 	{
+		ArchivedWAL_iterator iter;
+
 		if (!read_archive_file(privateInfo, XLOG_BLCKSZ))
 			pg_fatal("could not find WAL in archive \"%s\"",
 					 privateInfo->archive_name);
 
-		entry = privateInfo->cur_file;
+		ArchivedWAL_start_iterate(privateInfo->archive_wal_htab, &iter);
+		while ((entry = ArchivedWAL_iterate(privateInfo->archive_wal_htab,
+											&iter)) != NULL)
+		{
+			if (entry->read_len >= sizeof(XLogLongPageHeaderData))
+				break;
+		}
 	}
 
 	/* Extract the WAL segment size from the long page header */
diff --git a/src/bin/pg_waldump/pg_waldump.h b/src/bin/pg_waldump/pg_waldump.h
index cde7c6ca3f2..ca0dfd97168 100644
--- a/src/bin/pg_waldump/pg_waldump.h
+++ b/src/bin/pg_waldump/pg_waldump.h
@@ -44,7 +44,14 @@ typedef struct XLogDumpPrivate
 	Size		archive_read_buf_size;
 #endif
 
-	/* What the archive streamer is currently reading */
+	/*
+	 * The buffer for the WAL file the archive streamer is currently reading,
+	 * or NULL if none.  It is quite risky to examine this anywhere except in
+	 * astreamer_waldump_content(), since it can change multiple times during
+	 * a single read_archive_file() call.  However, it is safe to assume that
+	 * if cur_file is different from a particular ArchivedWALFile of interest,
+	 * then the archive streamer has finished reading that file.
+	 */
 	struct ArchivedWALFile *cur_file;
 
 	/*
-- 
2.43.0

From 9978359771c14b411704d808abc0f602119f0a9f Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 05:31:14 -0400
Subject: [PATCH v5 1/5] Fix finalization of decompressor astreamers.

Send the correct amount of data to the next astreamer, not the
whole allocated buffer size.  It's unclear how we missed this bug;
perhaps the use-cases so far are insensitive to trailing garbage.

Author: Andrew Dunstan <[email-masked]>

Discussion: https://postgr.es/m/[email-masked]
---
 src/fe_utils/astreamer_gzip.c | 9 +++++----
 src/fe_utils/astreamer_lz4.c  | 9 +++++----
 src/fe_utils/astreamer_zstd.c | 2 +-
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/src/fe_utils/astreamer_gzip.c b/src/fe_utils/astreamer_gzip.c
index 2e080c37a58..df392f67cab 100644
--- a/src/fe_utils/astreamer_gzip.c
+++ b/src/fe_utils/astreamer_gzip.c
@@ -347,10 +347,11 @@ astreamer_gzip_decompressor_finalize(astreamer *streamer)
 	 * End of the stream, if there is some pending data in output buffers then
 	 * we must forward it to next streamer.
 	 */
-	astreamer_content(mystreamer->base.bbs_next, NULL,
-					  mystreamer->base.bbs_buffer.data,
-					  mystreamer->base.bbs_buffer.maxlen,
-					  ASTREAMER_UNKNOWN);
+	if (mystreamer->bytes_written > 0)
+		astreamer_content(mystreamer->base.bbs_next, NULL,
+						  mystreamer->base.bbs_buffer.data,
+						  mystreamer->bytes_written,
+						  ASTREAMER_UNKNOWN);
 
 	astreamer_finalize(mystreamer->base.bbs_next);
 }
diff --git a/src/fe_utils/astreamer_lz4.c b/src/fe_utils/astreamer_lz4.c
index 2bc32b42879..605c188007b 100644
--- a/src/fe_utils/astreamer_lz4.c
+++ b/src/fe_utils/astreamer_lz4.c
@@ -397,10 +397,11 @@ astreamer_lz4_decompressor_finalize(astreamer *streamer)
 	 * End of the stream, if there is some pending data in output buffers then
 	 * we must forward it to next streamer.
 	 */
-	astreamer_content(mystreamer->base.bbs_next, NULL,
-					  mystreamer->base.bbs_buffer.data,
-					  mystreamer->base.bbs_buffer.maxlen,
-					  ASTREAMER_UNKNOWN);
+	if (mystreamer->bytes_written > 0)
+		astreamer_content(mystreamer->base.bbs_next, NULL,
+						  mystreamer->base.bbs_buffer.data,
+						  mystreamer->bytes_written,
+						  ASTREAMER_UNKNOWN);
 
 	astreamer_finalize(mystreamer->base.bbs_next);
 }
diff --git a/src/fe_utils/astreamer_zstd.c b/src/fe_utils/astreamer_zstd.c
index f26abcfd0fa..4b43ab795e3 100644
--- a/src/fe_utils/astreamer_zstd.c
+++ b/src/fe_utils/astreamer_zstd.c
@@ -347,7 +347,7 @@ astreamer_zstd_decompressor_finalize(astreamer *streamer)
 	if (mystreamer->zstd_outBuf.pos > 0)
 		astreamer_content(mystreamer->base.bbs_next, NULL,
 						  mystreamer->base.bbs_buffer.data,
-						  mystreamer->base.bbs_buffer.maxlen,
+						  mystreamer->zstd_outBuf.pos,
 						  ASTREAMER_UNKNOWN);
 
 	astreamer_finalize(mystreamer->base.bbs_next);
-- 
2.43.0

From 91ed0e69a7df00be147548973a8172322998b528 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 06:53:34 -0400
Subject: [PATCH v5 4/5] Fix get_archive_wal_entry to handle cur_file
 transitions reliably.

As noted by Tom Lane, get_archive_wal_entry() uses cur_file in an
unsafe way: a single read_archive_file() call can trigger multiple
astreamer callbacks when compression is effective, causing cur_file
to change several times (entry -> NULL -> new entry) within one
call.  The old code captured cur_file before the read and checked
for changes after, but this missed intermediate transitions.  This
could cause spill-file handles to leak or data to not be flushed
when the streamer finishes one segment and starts another within
the same read.

Restructure the spill logic to explicitly track the entry being
spilled (spill_entry) separately from cur_file, and detect
transitions at the top of each loop iteration.  Also ensure spill
file handles are closed on both success and error paths.

Discussion: https://postgr.es/m/[email-masked]
---
 src/bin/pg_waldump/archive_waldump.c | 117 ++++++++++++++++-----------
 1 file changed, 69 insertions(+), 48 deletions(-)

diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index 3fce2183099..93ed856c674 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -463,11 +463,18 @@ free_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
  * found.  If the archive streamer is reading a WAL file from the archive that
  * is not currently needed, that data is spilled to a temporary file for later
  * retrieval.
+ *
+ * Because a single read_archive_file() call may trigger multiple astreamer
+ * callbacks (especially when data compresses well), cur_file can change
+ * several times within one call: from one entry to NULL (member trailer),
+ * and then to a new entry (next member header).  The spill logic below
+ * handles this by flushing and closing per-entry state whenever we detect
+ * that the streamer has moved on.
  */
 static ArchivedWALFile *
 get_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
 {
-	ArchivedWALFile *entry = NULL;
+	ArchivedWALFile *spill_entry = NULL;
 	FILE	   *write_fp = NULL;
 
 	/*
@@ -477,6 +484,8 @@ get_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
 	 */
 	while (1)
 	{
+		ArchivedWALFile *entry;
+
 		/*
 		 * Search hash table.
 		 *
@@ -488,64 +497,76 @@ get_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
 		entry = ArchivedWAL_lookup(privateInfo->archive_wal_htab, fname);
 
 		if (entry != NULL)
+		{
+			/* Close any open spill file before returning. */
+			if (write_fp != NULL)
+				fclose(write_fp);
 			return entry;
-
-		/*
-		 * Capture the current entry before calling read_archive_file(),
-		 * because cur_file may advance to a new segment during streaming. We
-		 * hold this reference so we can flush any remaining buffer data and
-		 * close the write handle once we detect that cur_file has moved on.
-		 */
-		entry = privateInfo->cur_file;
-
-		/*
-		 * Fetch more data either when no current file is being tracked or
-		 * when its buffer has been fully flushed to the temporary file.
-		 */
-		if (entry == NULL || entry->buf->len == 0)
-		{
-			if (!read_archive_file(privateInfo, READ_CHUNK_SIZE))
-				break;			/* archive file ended */
 		}
 
 		/*
-		 * Archive streamer is reading a non-WAL file or an irrelevant WAL
-		 * file.
-		 */
-		if (entry == NULL)
-			continue;
-
-		/*
-		 * The streamer is producing a WAL segment that isn't the one asked
-		 * for; it must be arriving out of order.  Spill its data to disk so
-		 * it can be read back when needed.
-		 */
-		Assert(strcmp(fname, entry->fname) != 0);
-
-		/* Create a temporary file if one does not already exist */
-		if (!entry->spilled)
-		{
-			write_fp = prepare_tmp_write(entry->fname, privateInfo);
-			entry->spilled = true;
-		}
-
-		/* Flush data from the buffer to the file */
-		perform_tmp_write(entry->fname, entry->buf, write_fp);
-		resetStringInfo(entry->buf);
-
-		/*
-		 * If cur_file changed since we captured entry above, the archive
-		 * streamer has finished this segment and moved on.  Close its spill
-		 * file handle so data is flushed to disk before the next segment
-		 * starts writing to a different handle.
+		 * If the streamer has moved on to a different entry than the one we
+		 * were spilling, flush any remaining data for the old entry and close
+		 * its spill file.
 		 */
-		if (entry != privateInfo->cur_file && write_fp != NULL)
+		if (spill_entry != NULL && spill_entry != privateInfo->cur_file)
 		{
+			if (spill_entry->buf->len > 0)
+			{
+				perform_tmp_write(spill_entry->fname, spill_entry->buf,
+								  write_fp);
+				resetStringInfo(spill_entry->buf);
+			}
 			fclose(write_fp);
 			write_fp = NULL;
+			spill_entry = NULL;
 		}
+
+		/*
+		 * If no WAL file is currently being streamed (cur_file is NULL), or
+		 * the current spill entry's buffer has been fully flushed, we need
+		 * more data from the archive.
+		 */
+		if (privateInfo->cur_file == NULL ||
+			(spill_entry != NULL && spill_entry->buf->len == 0))
+		{
+			if (!read_archive_file(privateInfo, READ_CHUNK_SIZE))
+				break;			/* archive fully exhausted */
+			continue;			/* re-check hash table and cur_file */
+		}
+
+		/*
+		 * cur_file points to a WAL segment that isn't the one asked for; it
+		 * must be arriving out of order.  Spill its data to disk so it can be
+		 * read back when needed.
+		 */
+		spill_entry = privateInfo->cur_file;
+		Assert(strcmp(fname, spill_entry->fname) != 0);
+
+		/* Create a temporary file if one does not already exist */
+		if (!spill_entry->spilled)
+		{
+			write_fp = prepare_tmp_write(spill_entry->fname, privateInfo);
+			spill_entry->spilled = true;
+		}
+
+		/* Flush data from the buffer to the file */
+		perform_tmp_write(spill_entry->fname, spill_entry->buf, write_fp);
+		resetStringInfo(spill_entry->buf);
+
+		/*
+		 * Read more data from the archive.  This may add data to the current
+		 * spill_entry's buffer, advance cur_file to a new entry, or set
+		 * cur_file to NULL (member trailer).
+		 */
+		if (!read_archive_file(privateInfo, READ_CHUNK_SIZE))
+			break;				/* archive fully exhausted */
 	}
 
+	/* Close any open spill file before erroring out. */
+	if (write_fp != NULL)
+		fclose(write_fp);
+
 	/* Requested WAL segment not found */
 	pg_fatal("could not find WAL \"%s\" in archive \"%s\"",
 			 fname, privateInfo->archive_name);
-- 
2.43.0

From 0de5fd971602ae00fd3bd62cf5da0d8f0a3cce5a Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 05:32:45 -0400
Subject: [PATCH v5 2/5] Fix failure to finalize the decompression pipeline at
 archive EOF.

archive_waldump.c called astreamer_finalize() nowhere.  This meant
that any data retained in decompression buffers at the moment we
detect archive EOF would never reach astreamer_waldump_content(),
resulting in surprising failures if we actually need the last few
bytes of the archive file.

To fix, make read_archive_file() do the finalize once it detects
EOF.  Change its API to return a boolean "yes there's more data"
rather than the entirely-misleading raw count of bytes read.

Also document the contract that cur_file can change (or become NULL)
during a single read_archive_file() call, since the decompression
pipeline may produce enough output to trigger multiple astreamer
callbacks.

Author: Tom Lane <[email-masked]>

Discussion: https://postgr.es/m/[email-masked]
---
 src/bin/pg_waldump/archive_waldump.c | 50 +++++++++++++++++++++++-----
 src/bin/pg_waldump/pg_waldump.h      |  1 +
 2 files changed, 42 insertions(+), 9 deletions(-)

diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index b078c2d6960..cd092a057ef 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -89,7 +89,7 @@ typedef struct astreamer_waldump
 
 static ArchivedWALFile *get_archive_wal_entry(const char *fname,
 											  XLogDumpPrivate *privateInfo);
-static int	read_archive_file(XLogDumpPrivate *privateInfo, Size count);
+static bool read_archive_file(XLogDumpPrivate *privateInfo, Size count);
 static void setup_tmpwal_dir(const char *waldir);
 static void cleanup_tmpwal_dir_atexit(void);
 
@@ -139,6 +139,7 @@ init_archive_reader(XLogDumpPrivate *privateInfo,
 		pg_fatal("could not open file \"%s\"", privateInfo->archive_name);
 
 	privateInfo->archive_fd = fd;
+	privateInfo->archive_fd_eof = false;
 
 	streamer = astreamer_waldump_new(privateInfo);
 
@@ -178,7 +179,7 @@ init_archive_reader(XLogDumpPrivate *privateInfo,
 	 */
 	while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
 	{
-		if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
+		if (!read_archive_file(privateInfo, XLOG_BLCKSZ))
 			pg_fatal("could not find WAL in archive \"%s\"",
 					 privateInfo->archive_name);
 
@@ -236,9 +237,10 @@ free_archive_reader(XLogDumpPrivate *privateInfo)
 	/*
 	 * NB: Normally, astreamer_finalize() is called before astreamer_free() to
 	 * flush any remaining buffered data or to ensure the end of the tar
-	 * archive is reached.  However, when decoding WAL, once we hit the end
-	 * LSN, any remaining buffered data or unread portion of the archive can
-	 * be safely ignored.
+	 * archive is reached.  read_archive_file() may have done so.  However,
+	 * when decoding WAL we can stop once we hit the end LSN, so we may never
+	 * have read all of the input file.  In that case any remaining buffered
+	 * data or unread portion of the archive can be safely ignored.
 	 */
 	astreamer_free(privateInfo->archive_streamer);
 
@@ -384,7 +386,7 @@ read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
 						 fname, privateInfo->archive_name,
 						 (long long int) (count - nbytes),
 						 (long long int) count);
-			if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
+			if (!read_archive_file(privateInfo, READ_CHUNK_SIZE))
 				pg_fatal("unexpected end of archive \"%s\" while reading \"%s\": read %lld of %lld bytes",
 						 privateInfo->archive_name, fname,
 						 (long long int) (count - nbytes),
@@ -490,7 +492,7 @@ get_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
 		 */
 		if (entry == NULL || entry->buf->len == 0)
 		{
-			if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
+			if (!read_archive_file(privateInfo, READ_CHUNK_SIZE))
 				break;			/* archive file ended */
 		}
 
@@ -540,8 +542,22 @@ get_archive_wal_entry(const char *fname, XLogDumpPrivate *privateInfo)
 /*
  * Reads a chunk from the archive file and passes it through the streamer
  * pipeline for decompression (if needed) and tar member extraction.
+ *
+ * count is the maximum amount to try to read this time.  Note that it's
+ * measured in raw file bytes, and may have little to do with how much
+ * comes out of decompression/extraction.
+ *
+ * Returns true if successful, false if there is no more data.
+ *
+ * Callers must be aware that a single call may trigger multiple callbacks
+ * in astreamer_waldump_content, so privateInfo->cur_file can change value
+ * (or become NULL) during a call.  In particular, cur_file is set to NULL
+ * when the ASTREAMER_MEMBER_TRAILER callback fires at the end of a tar
+ * member; it is then set to a new entry when the next WAL member's
+ * ASTREAMER_MEMBER_HEADER callback fires, which may or may not happen
+ * within the same call.
  */
-static int
+static bool
 read_archive_file(XLogDumpPrivate *privateInfo, Size count)
 {
 	int			rc;
@@ -549,6 +565,11 @@ read_archive_file(XLogDumpPrivate *privateInfo, Size count)
 	/* The read request must not exceed the allocated buffer size. */
 	Assert(privateInfo->archive_read_buf_size >= count);
 
+	/* Fail if we already reached EOF in a prior call. */
+	if (privateInfo->archive_fd_eof)
+		return false;
+
+	/* Try to read some more data. */
 	rc = read(privateInfo->archive_fd, privateInfo->archive_read_buf, count);
 	if (rc < 0)
 		pg_fatal("could not read file \"%s\": %m",
@@ -562,8 +583,19 @@ read_archive_file(XLogDumpPrivate *privateInfo, Size count)
 		astreamer_content(privateInfo->archive_streamer, NULL,
 						  privateInfo->archive_read_buf, rc,
 						  ASTREAMER_UNKNOWN);
+	else
+	{
+		/*
+		 * We reached EOF, but there is probably still data queued in the
+		 * astreamer pipeline's buffers.  Flush it out to ensure that we
+		 * process everything.
+		 */
+		astreamer_finalize(privateInfo->archive_streamer);
+		/* Set flag to ensure we don't finalize more than once. */
+		privateInfo->archive_fd_eof = true;
+	}
 
-	return rc;
+	return true;
 }
 
 /*
diff --git a/src/bin/pg_waldump/pg_waldump.h b/src/bin/pg_waldump/pg_waldump.h
index 36893624f53..cde7c6ca3f2 100644
--- a/src/bin/pg_waldump/pg_waldump.h
+++ b/src/bin/pg_waldump/pg_waldump.h
@@ -35,6 +35,7 @@ typedef struct XLogDumpPrivate
 	char	   *archive_dir;
 	char	   *archive_name;	/* Tar archive filename */
 	int			archive_fd;		/* File descriptor for the open tar file */
+	bool		archive_fd_eof; /* Have we reached EOF on archive_fd? */
 
 	astreamer  *archive_streamer;
 	char	   *archive_read_buf;	/* Reusable read buffer for archive I/O */
-- 
2.43.0

From cab3e116c1064afcc16e638555c74f7f460be55b Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 06:53:58 -0400
Subject: [PATCH v5 5/5] Remove nonfunctional tar file trailer size check.

The ASTREAMER_ARCHIVE_TRAILER case in astreamer_tar_parser_content()
intended to reject tar files whose trailer exceeded 2 blocks.  However,
the check compared 'len' after astreamer_buffer_bytes() had already
consumed all the data and set len to 0, so the pg_fatal() could never
fire.

Moreover, per the POSIX specification for the ustar format, the last
physical block of a tar archive is always full-sized, and "logical
records after the two zero logical records may contain undefined data."
GNU tar, for example, zero-pads its output to a 10kB boundary by
default.  So rejecting extra data after the two zero blocks would be
wrong even if the check worked.

Remove the dead check and update the comment to explain why trailing
data is expected and harmless.

Per report from Tom Lane.

Discussion: https://postgr.es/m/[email-masked]
---
 src/fe_utils/astreamer_tar.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/src/fe_utils/astreamer_tar.c b/src/fe_utils/astreamer_tar.c
index 4b187f0a8c4..f8be5e4ff8a 100644
--- a/src/fe_utils/astreamer_tar.c
+++ b/src/fe_utils/astreamer_tar.c
@@ -236,12 +236,16 @@ astreamer_tar_parser_content(astreamer *streamer, astreamer_member *member,
 
 				/*
 				 * We've seen an end-of-archive indicator, so anything more is
-				 * buffered and sent as part of the archive trailer. But we
-				 * don't expect more than 2 blocks.
+				 * buffered and sent as part of the archive trailer.
+				 *
+				 * Per POSIX, the last physical block of a tar archive is
+				 * always full-sized, so there may be undefined data after the
+				 * two zero blocks that mark end-of-archive.  GNU tar, for
+				 * example, zero-pads to a 10kB boundary by default.  We just
+				 * buffer whatever we receive and pass it along at finalize
+				 * time.
 				 */
 				astreamer_buffer_bytes(streamer, &data, &len, len);
-				if (len > 2 * TAR_BLOCK_SIZE)
-					pg_fatal("tar file trailer exceeds 2 blocks");
 				return;
 
 			default:
-- 
2.43.0

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 22 Mar 2026 17:19:09 -0400

On Sun, Mar 22, 2026 at 2:17 PM Tom Lane <[email-masked]> wrote:

> I wrote:
> > ... We can make this function far simpler
> > and more obviously correct if we just accept that we'll read a
> > WAL file completely before spilling it.  See my proposed
> > alternative to 0004, attached.
>
> Actually, we can make that better yet by not expecting
> get_archive_wal_entry to clean up after init_archive's
> failure to free all irrelevant hashtable entries.
> Better version attached.
>
>
>

Yeah, this looks good.  I know we also still need to do something about
rmtree trying to remove files we haven't closed. But what we have so far in
this set LGTM. If you want to push this I'm good, otherwise I'll look at it
tomorrow or Tuesday.

cheers

andrew

---

From: Michael Paquier <[email-masked]>
Date: Tue, 24 Mar 2026 12:11:44 +0900

On Sun, Mar 22, 2026 at 11:02:20PM -0400, Tom Lane wrote:
> Proposed patch attached.  There might be an argument for using some
> other size than 256K for the other two decompressors, but my
> inclination is to try to make all three use roughly the same block
> size.  (See also 66ec01dc4.)

The buildfarm has switched mostly to green, except on this one:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42
--
Michael

---

From: Andres Freund <[email-masked]>
Date: Wed, 25 Mar 2026 13:28:26 -0400

Hi,

> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42

I think there's a few more failues. Fairywren regularly fails, including in a
run from today.

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2026-03-25%2003%3A48%3A06

There also are a lot of CI failures. E.g.

https://cirrus-ci.com/task/6153854431002624
https://api.cirrus-ci.com/v1/artifact/task/6153854431002624/testrun/build/testrun/pg_waldump/001_basic/log/regress_log_001_basic
# Running: pg_waldump --path C:\msys64\tmp\tNfU5IfQ4a/pg_wal.tar.gz --start 0/01806F48 --end 0/03093BD8
[22:46:25.358](3.991s) not ok 160 - runs with path option and start and end locations: exit code 0
[22:46:25.363](0.005s) #   Failed test 'runs with path option and start and end locations: exit code 0'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.364](0.001s) ok 161 - runs with path option and start and end locations: no stderr
[22:46:25.365](0.001s) not ok 162 - runs with path option and start and end locations: matches
[22:46:25.365](0.000s) #   Failed test 'runs with path option and start and end locations: matches'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.366](0.000s) #                   ''
#     doesn't match '(?^:.)'

I was first suspecting that this is due to

commit 1c162c965a1
Author: Fujii Masao <[email-masked]>
Date:   2026-03-24 22:33:09 +0900

    Report detailed errors from XLogFindNextRecord() failures.

but there are afaict failures from before that:
https://cirrus-ci.com/task/5501609960013824
which is for 4019f725f5d, preceding 1c162c965a1
and
https://cirrus-ci.com/task/5317196043255808

It does feel however the failure frequency has increased substantially:
https://cirrus-ci.com/github/postgres/postgres/master

Greetings,

Andres Freund

---

From: Thomas Munro <[email-masked]>
Date: Sun, 29 Mar 2026 10:36:06 +1300

> I think there's a few more failues. Fairywren regularly fails, including in a

This fails 100% of the time on my machine, even after e9d72348 and ff84efe4, eg:

# Running: pg_waldump --path /tmp/D8WG1Sv2HE/pg_wal.tar --start
0/017A2610 --end 0/02093848
[09:43:29.288](0.148s) not ok 104 - runs with path option and start
and end locations: exit code 0
[09:43:29.289](0.001s) #   Failed test 'runs with path option and
start and end locations: exit code 0'
#   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
line 402.
[09:43:29.290](0.001s) not ok 105 - runs with path option and start
and end locations: no stderr
[09:43:29.291](0.001s) #   Failed test 'runs with path option and
start and end locations: no stderr'
#   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
line 402.
[09:43:29.291](0.000s) #          got: 'pg_waldump: error: could not
find WAL "000000010000000000000002" in archive "pg_wal.tar"
# '

I can see that it is wrong about the contents of the tar file:

$ pg_waldump --path _tmp_H_1gv81G1L_pg_wal.tar --start 0/017A2610
--end 0/020934F8 2>&1 | tail -3
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFC1B8, prev 0/01FFC178, desc: INSERT off 40, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Transaction len (rec/tot):     46/    46, tx:        720, lsn:
0/01FFC200, prev 0/01FFC1B8, desc: COMMIT 2026-03-29 10:15:24.112967
NZDT
pg_waldump: error: could not find WAL "000000010000000000000002" in
archive "_tmp_H_1gv81G1L_pg_wal.tar"

$ tar tvf _tmp_H_1gv81G1L_pg_wal.tar
drwx------  0 tmunro tmunro      0 Mar 29 10:15 archive_status/
-rw-------  0 tmunro tmunro      0 Mar 29 10:15
archive_status/000000010000000000000002.ready
-rw-------  0 tmunro tmunro      0 Mar 29 10:15
archive_status/000000010000000000000001.ready
drwx------  0 tmunro tmunro      0 Mar 29 10:08 summaries/
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000002
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000001
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000003

It seems like the place we'd be looking for the file is in
astreamer_tar_header(), so I added in some caveman debugging:

    /*
     * Parse key fields out of the header.
     */
fprintf(stderr, "XXXX [%s] XXXX\n", &buffer[TAR_OFFSET_NAME]);
    strlcpy(member->pathname, &buffer[TAR_OFFSET_NAME], MAXPGPATH);
    if (member->pathname[0] == '\0')
        pg_fatal("tar member has empty name");

Now I see:

XXXX [archive_status/] XXXX
XXXX [archive_status/000000010000000000000002.ready] XXXX
XXXX [archive_status/000000010000000000000001.ready] XXXX
XXXX [summaries/] XXXX
XXXX [PaxHeader/000000010000000000000002] XXXX
XXXX [GNUSparseFile.0/000000010000000000000002] XXXX
XXXX [000000010000000000000001] XXXX
rmgr: XLOG        len (rec/tot):     30/    30, tx:          0, lsn:
0/017A2610, prev 0/017A25F0, desc: NEXTOID 24576
rmgr: Standby     len (rec/tot):     42/    42, tx:        692, lsn:
0/017A2630, prev 0/017A2610, desc: LOCK xid 692 db 5 rel 16384
rmgr: Storage     len (rec/tot):     42/    42, tx:        692, lsn:
0/017A2660, prev 0/017A2630, desc: CREATE base/5/16384
... lots more normal output ...
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFBED8, prev 0/01FFBE98, desc: INSERT off 97, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Heap        len (rec/tot):    575/   575, tx:        720, lsn:
0/01FFBF20, prev 0/01FFBED8, desc: INSERT off: 12, flags: 0x08, blkref
#0: rel 1663/5/16393 blk 52
rmgr: Btree       len (rec/tot):     64/    64, tx:        720,
lsn:XXXX [PaxHeader/000000010000000000000003] XXXX
XXXX [GNUSparseFile.0/000000010000000000000003] XXXX
 0/01FFC178, prev 0/01FFBF20, desc: INSERT_LEAF off: 344, blkref #0:
rel 1663/5/16396 blk 2
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFC1B8, prev 0/01FFC178, desc: INSERT off 40, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Transaction len (rec/tot):     46/    46, tx:        720, lsn:
0/01FFC200, prev 0/01FFC1B8, desc: COMMIT 2026-03-29 10:15:24.112967
NZDT
pg_waldump: error: could not find WAL "000000010000000000000002" in
archive "_tmp_H_1gv81G1L_pg_wal.tar"

Seems like it already stepped over 000000010000000000000002 earlier?
Could it be a table-of-contents order dependency bug or something like
that?

---

From: Thomas Munro <[email-masked]>
Date: Sun, 29 Mar 2026 11:08:43 +1300

On Sun, Mar 29, 2026 at 10:49 AM Tom Lane <[email-masked]> wrote:
> Huh.  What is your platform exactly?  Maybe more to the point,
> what is the tar program you're using?

FreeBSD tar 3.8.2.

> If you look at the TAP script, you'll see that it tries to randomize
> the order of the entries in the tar file (see sub generate_archive).
> So if that's the problem, it shouldn't reproduce 100%, and also we
> should be seeing lots of freckles on the buildfarm.  We're not, so
> there must be something off-the-beaten-track about your test
> environment.

Right, I see now.  There is something different about
000000010000000000000002 though: it doesn't seem to have a normal
(non-PAX, non-GNU) TOC entry, unlike000000010000000000000001.  Trying
to figure out why...

---

From: Tomas Vondra <[email-masked]>
Date: Sat, 28 Mar 2026 23:57:37 +0100

On 3/28/26 23:36, Tom Lane wrote:
> I wrote:
>> However ... I do not find any indication in the GNU tar docs
>> that it produces sparse files by default.  It looks like you
>> need to say -S/--sparse to make that happen.  Maybe you have
>> a version that's been hacked to make that the default?
> 
> Bleah.  Digging in the man pages at freebsd.org, I read
> 
>     --read-sparse
> 	       (c, r, u	modes only) Read sparse	file  information  from	 disk.
> 	       This  is	the reverse of --no-read-sparse	and the	default	behav-
> 	       ior.
> 
> It's apparently been there and been default since FreeBSD 13.1.
> This leads one to wonder how come BF member dikkop is managing
> to run this test successfully.  I speculate that it's using a
> filesystem type that doesn't do sparse files (cc'ing Vondra
> for confirmation on that).
> 

It's running on ufs. But I think the explanation is very simple. We had
a short power outage on Thursday, and the FreeBSD machine failed to boot
properly after the power was restored. IIUC this test is new, right?

I fixed the machine, it'll start running the tests in a couple minutes.

regards

-- 
Tomas Vondra

---

From: Thomas Munro <[email-masked]>
Date: Sun, 29 Mar 2026 12:15:39 +1300

On Sun, Mar 29, 2026 at 11:37 AM Tom Lane <[email-masked]> wrote:
> It looks like to make this test stable on modern FreeBSD,
> we need to see if tar accepts --no-read-sparse and use that
> switch if so.

Yeah.  Here's my attempt at perl.

I think your Mac probably has a similar tar program BTW... but apfs
probably doesn't go around making holes visible to lseek()
automatically or at least as eagerly as my ZFS system.
From b1a9131d072dadfe9c5666e75a932ba70b8d2f99 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email-masked]>
Date: Sun, 29 Mar 2026 12:03:42 +1300
Subject: [PATCH] Fix pg_waldump test for libarchive tar.

libarchive tar (the one shipped on macOS, *BSD systems) might decide to
archive non-standard sparse encoding with GNU extensions (unlike GNU tar
itself) by default.  pg_waldump can't read them.  Suppress that, if $TAR
understands --no-sparse-files.

Discussion: https://postgr.es/m/1624716.1774736283%40sss.pgh.pa.us
---
 src/bin/pg_waldump/t/001_basic.pl | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 8bb8fa225f6..d911296bb66 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -11,6 +11,14 @@ use Test::More;
 use List::Util qw(shuffle);
 
 my $tar = $ENV{TAR};
+my @TAR_C_FLAGS;
+
+# libarchive tar (as found on *BSD and macOS) might create sparse files by
+# default, and we can't read them
+if (system("$tar --no-read-sparse -c - /dev/null > /dev/null") == 0)
+{
+  push(@TAR_C_FLAGS, "--no-read-sparse");
+}
 
 program_help_ok('pg_waldump');
 program_version_ok('pg_waldump');
@@ -331,7 +339,6 @@ sub test_pg_waldump
 sub generate_archive
 {
 	my ($archive, $directory, $compression_flags) = @_;
-
 	my @files;
 	opendir my $dh, $directory or die "opendir: $!";
 	while (my $entry = readdir $dh) {
@@ -346,7 +353,7 @@ sub generate_archive
 	# move into the WAL directory before archiving files
 	my $cwd = getcwd;
 	chdir($directory) || die "chdir: $!";
-	command_ok([$tar, $compression_flags, $archive, @files]);
+	command_ok([$tar, @TAR_C_FLAGS, $compression_flags, $archive, @files]);
 	chdir($cwd) || die "chdir: $!";
 }
 
-- 
2.52.0

---

From: Tomas Vondra <[email-masked]>
Date: Sun, 29 Mar 2026 15:33:10 +0200

On 3/29/26 00:12, Tom Lane wrote:
> Tomas Vondra <[email-masked]> writes:
> Not that new, it dates to b15c15139, about a week ago.
> 
> I've reproduced Thomas' failure on a local FreeBSD 15.0 image
> using zfs, and confirmed that this cowboy hack fixes it:
> 

Interesting. Then I guess it has to be due to some difference in ufs vs.
zfs, when handling sparse files. It might be useful to add a bit more
variation here, and switch some of the animals to non-default
filesystems (not just the FreeBSD ones, which we seem to have only two
that run reasonably often). I'd bet most of the linux systems run on
ext4/xfs, few on btrfs/zfs.

regards

-- 
Tomas Vondra

---

From: Thomas Munro <[email-masked]>
Date: Mon, 30 Mar 2026 11:11:50 +1300

> ext4/xfs, few on btrfs/zfs.

UFS does have sparse files (its ancestor invented them some time
around (time_t) 0), it just doesn't make them unless you tell it to.
PostgreSQL only does that if you set wal_init_zero=false.

ZFS is different because it creates holes automagically when you write
zeroes, at least if compression is enabled so it has to scan all your
bytes anyway.

I was curious to know if BTRFS does that too, or hides
zero-compression at some lower invisible level:

$ echo "hello" > 1MB-sparse.dat
$ truncate -s 512KB 1MB-sparse.dat
$ echo "world" >> 1MB-sparse.dat
$ truncate -s 1MB 1MB-sparse.dat
$ ls -l 1MB-sparse.dat
-rw-rw-r-- 1 tmunro tmunro 1000000 Mar 30 10:11 1MB-sparse.dat
$ du -hs 1MB-sparse.dat
8.0K    1MB-sparse.dat
$ strace tar -S -cf foo.tar 1MB-sparse.dat 2>&1 | grep seek
lseek(4, 0, SEEK_DATA)                  = 0
lseek(4, 0, SEEK_HOLE)                  = 4096
lseek(4, 4096, SEEK_DATA)               = 512000
lseek(4, 512000, SEEK_HOLE)             = 516096
lseek(4, 516096, SEEK_DATA)             = -1 ENXIO (No such device or address)

... so that's a yes, lseek sees holes that we didn't ask it to make,
just like on ZFS, but the rest of this trace of GNU tar -S -cf is
interesting:

lseek(5, 0, SEEK_SET)                   = 0
lseek(5, 0, SEEK_SET)                   = 0
lseek(4, 0, SEEK_SET)                   = 0
lseek(4, 512000, SEEK_SET)              = 512000
lseek(4, 1000000, SEEK_SET)             = 1000000

It didn't write out PAX format!  Instead it replicated the holes into
the tar file itself with SEEK_SET.

$ strings foo.tar | grep Sparse

You have to add --format=posix to enable the GNU behaviour that BSD
tar is emulating by default:

$ tar --format=posix -S -cf foo.tar 1MB-sparse.dat
$ strings foo.tar | grep Sparse
./GNUSparseFile.4190/1MB-sparse.dat

I expected GNU tar to be forced to do that if writing to non-seekable
output, eg "tar -S -c 1MB-sparse.dat | cat > foo.tar", but somehow it
manages to write out only ~10KB of plain ustar format that it is able
to restore to the full 1MB apparent size using some other trick, but
... ENOTIME, I dunno how it's doing that.  Might be interesting to see
if pg_waldump can read it though, 'cause the bytes aren't all there.

BTW I confirmed that Apple tar does have -S by default too, it's just
that APFS doesn't make holes magically, so this test would presumably
have broken on a Mac if wal_init_zero had been forced to zero (not
tested).

Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
unlikely to hit this in the wild, and the symptom is a confusing error
in a maintenance tool, not corruption, so I don't think this is a big
deal.  I might still try teaching the astreamer code to understand PAX
1.0 when it sees it in the next cycle though, for the benefit of
FreeBSD users.  A quick and dirty version could probably just unmangle
the name and skip the first block of data, since any valid WAL file
will not begin with a hole and valid WAL data will end at the first
hole and fail our verification, but of course a real implementation
should read the map properly[1]...

[1] https://www.gnu.org/software/tar/manual/html_node/PAX-1.html

---

From: Thomas Munro <[email-masked]>
Date: Mon, 30 Mar 2026 11:20:24 +1300

> ... so that's a yes, lseek sees holes that we didn't ask it to make,

Oops, sorry, I wrote that email too fast and got my examples mixed up,
BTFS actually *doesnt* do that automatically, that was of course a
trace showing a file with explicitly made holes.  So this is probably
be a ZFS-only issue unless you're using wal_init_zero=0, and then any
file system could result in PAX-sparse-format tarballs, but even then
only if you use non-default switches that in practice no one will use
with GNU tar, or if you use BSD tar.  So in practice this is a
FreeBSD-only issue.