public inbox for [email protected]
help / color / mirror / Atom feedFrom: Thomas Munro <[email protected]>
To: Tom Lane <[email protected]>
Cc: Tomas Vondra <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: Amul Sul <[email protected]>
Cc: Zsolt Parragi <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Chao Li <[email protected]>
Cc: Anthonin Bonnefoy <[email protected]>
Cc: Fujii Masao <[email protected]>
Cc: Jakub Wartak <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
Date: Sun, 29 Mar 2026 12:15:39 +1300
Message-ID: <CA+hUKGK-GaHRh5mO0WKWO6MiL-iJ=+j_vVg4f-3pQXATNfRC2Q@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<CAD5tBcLVWKnph3iB-VPuPKR0dCckOJRFZW2-4H7HTTmhw8-vOg@mail.gmail.com>
<[email protected]>
<[email protected]!!.pa.us>
<CAD5tBcLsYDz+Nzx8MryjxiKaN3fGKd4ZgXuN1Jn=CYxw9dh+AA@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
<CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
<[email protected]>
<[email protected]>
On Sun, Mar 29, 2026 at 11:37 AM Tom Lane <[email protected]> wrote:
> I wrote:
> > However ... I do not find any indication in the GNU tar docs
> > that it produces sparse files by default. It looks like you
> > need to say -S/--sparse to make that happen. Maybe you have
> > a version that's been hacked to make that the default?
>
> Bleah. Digging in the man pages at freebsd.org, I read
>
> --read-sparse
> (c, r, u modes only) Read sparse file information from disk.
> This is the reverse of --no-read-sparse and the default behav-
> ior.
>
> It's apparently been there and been default since FreeBSD 13.1.
> This leads one to wonder how come BF member dikkop is managing
> to run this test successfully. I speculate that it's using a
> filesystem type that doesn't do sparse files (cc'ing Vondra
> for confirmation on that).
>
> It looks like to make this test stable on modern FreeBSD,
> we need to see if tar accepts --no-read-sparse and use that
> switch if so.
Yeah. Here's my attempt at perl.
I think your Mac probably has a similar tar program BTW... but apfs
probably doesn't go around making holes visible to lseek()
automatically or at least as eagerly as my ZFS system.
Attachments:
[text/x-patch] 0001-Fix-pg_waldump-test-for-libarchive-tar.patch (1.8K, 2-0001-Fix-pg_waldump-test-for-libarchive-tar.patch)
download | inline diff:
From b1a9131d072dadfe9c5666e75a932ba70b8d2f99 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Sun, 29 Mar 2026 12:03:42 +1300
Subject: [PATCH] Fix pg_waldump test for libarchive tar.
libarchive tar (the one shipped on macOS, *BSD systems) might decide to
archive non-standard sparse encoding with GNU extensions (unlike GNU tar
itself) by default. pg_waldump can't read them. Suppress that, if $TAR
understands --no-sparse-files.
Discussion: https://postgr.es/m/1624716.1774736283%40sss.pgh.pa.us
---
src/bin/pg_waldump/t/001_basic.pl | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 8bb8fa225f6..d911296bb66 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -11,6 +11,14 @@ use Test::More;
use List::Util qw(shuffle);
my $tar = $ENV{TAR};
+my @TAR_C_FLAGS;
+
+# libarchive tar (as found on *BSD and macOS) might create sparse files by
+# default, and we can't read them
+if (system("$tar --no-read-sparse -c - /dev/null > /dev/null") == 0)
+{
+ push(@TAR_C_FLAGS, "--no-read-sparse");
+}
program_help_ok('pg_waldump');
program_version_ok('pg_waldump');
@@ -331,7 +339,6 @@ sub test_pg_waldump
sub generate_archive
{
my ($archive, $directory, $compression_flags) = @_;
-
my @files;
opendir my $dh, $directory or die "opendir: $!";
while (my $entry = readdir $dh) {
@@ -346,7 +353,7 @@ sub generate_archive
# move into the WAL directory before archiving files
my $cwd = getcwd;
chdir($directory) || die "chdir: $!";
- command_ok([$tar, $compression_flags, $archive, @files]);
+ command_ok([$tar, @TAR_C_FLAGS, $compression_flags, $archive, @files]);
chdir($cwd) || die "chdir: $!";
}
--
2.52.0
view thread (85+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
In-Reply-To: <CA+hUKGK-GaHRh5mO0WKWO6MiL-iJ=+j_vVg4f-3pQXATNfRC2Q@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox