MIME-Version: 1.0
References: <2250061.1774104346@sss.pgh.pa.us>
 <3341199.1774221191@sss.pgh.pa.us>
 <3424809.1774234940@sss.pgh.pa.us> <acIA8KKBTv-T53YG@paquier.xyz>
 <x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
 <CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
 <1624716.1774736283@sss.pgh.pa.us> <1626907.1774737417@sss.pgh.pa.us>
 <97a382c0-1f19-4ea0-951f-e37e6abc34a3@vondra.me>
 <1630755.1774739531@sss.pgh.pa.us>
 <c3a2b730-9d90-4a66-ae5e-50f6d77922d6@vondra.me>
 <CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
 <1873141.1774823011@sss.pgh.pa.us>
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
 <3049460.1775067940@sss.pgh.pa.us> <3118179.1775092964@sss.pgh.pa.us>
 <CA+hUKG+-pn14s_tjEBO6YKHmc=uRhGVn=w2oM91KKnEUc7pH0Q@mail.gmail.com>
 <3565835.1775147392@sss.pgh.pa.us> <3579709.1775151816@sss.pgh.pa.us>
 <63de1553-829a-488d-8ee0-976afb8dd32c@vondra.me>
 <3586483.1775155672@sss.pgh.pa.us>
 <CA+hUKG+A-=BKP69ze5oc4O5RSBZtKk7dGoqYLeOUQSo8xugTyg@mail.gmail.com>
 <3676229.1775170250@sss.pgh.pa.us>
In-Reply-To: <3676229.1775170250@sss.pgh.pa.us>
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 3 Apr 2026 12:49:21 +1300
Message-ID: 
 <CA+hUKGKfti_FMFuduXEZs96W5Boce9gSLZ5Ei158dFiuLuWLgA@mail.gmail.com>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>,
	Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>,
 Amul Sul <sulamul@gmail.com>,
	Zsolt Parragi <zsolt.parragi@percona.com>,
 Robert Haas <robertmhaas@gmail.com>,
	Chao Li <li.evan.chao@gmail.com>,
	Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>,
 Fujii Masao <masao.fujii@gmail.com>,
	Jakub Wartak <jakub.wartak@enterprisedb.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/mixed; boundary="000000000000bfde3d064e82db80"
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BhUKGKfti_FMFuduXEZs96W5Boce9gSLZ5Ei158dFiuLuWLgA%40mail.gmail.com>
Precedence: bulk

--000000000000bfde3d064e82db80
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 3, 2026 at 11:50=E2=80=AFAM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > How about using --format=3Dustar, instead of that sparse control stuff?
>
> I did it that way for GNU tar, but did not research whether bsdtar
> will take that option.  Feel free to hack on ebba64c08 some more.
>
> (It seems though that the two tars' locutions for "write to stdout"
> are different, so we might have to have separate tests even if they
> end up pushing the same option.)

I have:

$ tar --version
bsdtar 3.8.2 - libarchive 3.8.2 zlib/1.3.1 liblzma/5.8.1 libzstd/1.5.2
openssl/3.5.4 libb2/bundled
$ gtar --version
tar (GNU tar) 1.35
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.ht=
ml>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

This seems to work for both:

$ tar --format=3Dustar -c /dev/null  > /dev/null
tar: Removing leading '/' from member names
$ gtar --format=3Dustar -c /dev/null  > /dev/null
gtar: Removing leading `/' from member names

The attached passes with both, and regress_log_001_basic looks like:

# Running: /usr/bin/tar --format=3Dustar -cf /tmp/J_ifbfUOSd/pg_wal.tar
archive_status 000000010000000000000001 000000010000000000000003
000000010000000000000002 summaries
[12:12:24.301](0.072s) ok 101

# Running: /usr/local/bin/gtar --format=3Dustar -cf
/tmp/pbdsHdrAdw/pg_wal.tar 000000010000000000000002 archive_status
000000010000000000000003 summaries 000000010000000000000001
[12:18:14.739](0.050s) ok 101

I think a Windows system could be using either.  BSD tar comes
pre-installed by Microsoft and people often install GNU tools.  So I
think we should use File::Spec->devnull() instead of /dev/null, and
Andrew showed that working.  I doubt Windows is capable of making
sparse files (except perhaps with ReFS?), but it's nice to use the
same code everywhere and future-proof in case GNU carries out its
thread to switch to pax by default.  Windows probably has file
attributes that ustar can't represent (?), so I guess that might
motivate it to use pax headers if they are indeed added only when
needed.

Longer term I think we need to tolerate but ignore pax headers.  If I
understand the spirit of this long evolution, pax archives are
intended to be acceptable to pre-pax implementations, which implies
that they can't really change the meaning of the bits of the file
contents.  That's why GNU's --sparse hides funky file encodings from
old tars by renaming them to GNUSparseFile.%p/%f, and that leads back
to my original suggestion that we should figure out how to detect and
reject pax only if we failed to find the file under the expected name.
(Or of course we could just implement support for that, and I have a
half-baked trial patch for that but now is not the time.)

--000000000000bfde3d064e82db80
Content-Type: text/x-patch; charset="US-ASCII";
	name="0001-Harmonize-tar-option-tests-from-ebba64c0.patch"
Content-Disposition: attachment;
	filename="0001-Harmonize-tar-option-tests-from-ebba64c0.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mni40si00>
X-Attachment-Id: f_mni40si00

RnJvbSAwNjM1YzAxMDY5NDZjZTc2YzFiYjg0YTVjMTdiNzFkMGIwZTU3NGY3IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgTXVucm8gPHRob21hcy5tdW5yb0BnbWFpbC5jb20+
CkRhdGU6IEZyaSwgMyBBcHIgMjAyNiAxMjowMzo1NiArMTMwMApTdWJqZWN0OiBbUEFUQ0hdIEhh
cm1vbml6ZSB0YXIgb3B0aW9uIHRlc3RzIGZyb20gZWJiYTY0YzAuCgoqICBHTlUgYW5kIEJTRCB0
YXIgYm90aCB1bmRlcnN0YW5kIC0tZm9ybWF0PXVzdGFyLgoqICBXaW5kb3dzIGxhY2tzIC9kZXYv
bnVsbCwgYnV0IHBlcmwga25vd3MgaXRzIGxvY2FsIG5hbWUuCgpEaXNjdXNzaW9uOiBodHRwczov
L3Bvc3Rnci5lcy9tLzM2NzYyMjkuMTc3NTE3MDI1MCU0MHNzcy5wZ2gucGEudXMKQmFja3BhdGNo
LXRocm91Z2g6IDE4Ci0tLQogc3JjL3Rlc3QvcGVybC9Qb3N0Z3JlU1FML1Rlc3QvVXRpbHMucG0g
fCAxNSArKysrLS0tLS0tLS0tLS0KIDEgZmlsZSBjaGFuZ2VkLCA0IGluc2VydGlvbnMoKyksIDEx
IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3NyYy90ZXN0L3BlcmwvUG9zdGdyZVNRTC9UZXN0
L1V0aWxzLnBtIGIvc3JjL3Rlc3QvcGVybC9Qb3N0Z3JlU1FML1Rlc3QvVXRpbHMucG0KaW5kZXgg
MTIwOTk5ZjZhYzkuLjA1ZTE2OThlZmE2IDEwMDY0NAotLS0gYS9zcmMvdGVzdC9wZXJsL1Bvc3Rn
cmVTUUwvVGVzdC9VdGlscy5wbQorKysgYi9zcmMvdGVzdC9wZXJsL1Bvc3RncmVTUUwvVGVzdC9V
dGlscy5wbQpAQCAtMTMyOCwyMSArMTMyOCwxNCBAQCBzdWIgdGFyX3BvcnRhYmlsaXR5X29wdGlv
bnMKIAogCSMgR05VIHRhciB0eXBpY2FsbHkgcHJvZHVjZXMgZ251LWZvcm1hdCBhcmNoaXZlcywg
d2hpY2ggd2UgY2FuIHJlYWQgZmluZS4KIAkjIEJ1dCBzb21lIHBsYXRmb3JtcyBjb25maWd1cmUg
aXQgdG8gZGVmYXVsdCB0byBwb3NpeC9wYXggZm9ybWF0LCBhbmQKLQkjIGFwcGFyZW50bHkgdGhl
eSBlbmFibGUgLS1zcGFyc2UgdG9vLiAgT3ZlcnJpZGUgdGhhdC4KLQlpZiAoc3lzdGVtKCIkdGFy
IC0tZm9ybWF0PXVzdGFyIC1jIC1PIC9kZXYvbnVsbCA+L2Rldi9udWxsIDI+L2Rldi9udWxsIikK
KwkjIGFwcGFyZW50bHkgdGhleSBlbmFibGUgLS1zcGFyc2UgdG9vLiAgQlNEIHRhciBkb2VzIHNv
bWV0aGluZyBzaW1pbGFyLgorCSMgT3ZlcnJpZGUgdGhhdC4KKwlteSAkZGV2bnVsbCA9IEZpbGU6
OlNwZWMtPmRldm51bGwoKTsKKwlpZiAoc3lzdGVtKCIkdGFyIC0tZm9ybWF0PXVzdGFyIC1jICRk
ZXZudWxsID4kZGV2bnVsbCAyPiRkZXZudWxsIikKIAkJPT0gMCkKIAl7CiAJCXB1c2goQHRhcl9w
X2ZsYWdzLCAiLS1mb3JtYXQ9dXN0YXIiKTsKIAl9Ci0KLQkjIGJzZHRhciBhbHNvIGFyY2hpdmVz
IHNwYXJzZSBmaWxlcyBieSBkZWZhdWx0LCBidXQgaXQgc3BlbGxzIHRoZSBzd2l0Y2gKLQkjIHRv
IGRpc2FibGUgdGhhdCBkaWZmZXJlbnRseS4KLQlpZiAoc3lzdGVtKCIkdGFyIC0tbm8tcmVhZC1z
cGFyc2UgLWMgLSAvZGV2L251bGwgPi9kZXYvbnVsbCAyPi9kZXYvbnVsbCIpCi0JCT09IDApCi0J
ewotCQlwdXNoKEB0YXJfcF9mbGFncywgIi0tbm8tcmVhZC1zcGFyc2UiKTsKLQl9Ci0KIAlyZXR1
cm4gQHRhcl9wX2ZsYWdzOwogfQogCi0tIAoyLjUzLjAKCg==
--000000000000bfde3d064e82db80--