MIME-Version: 1.0
References: <2250061.1774104346@sss.pgh.pa.us>
 <2555285.1774131847@sss.pgh.pa.us>
 <2609460.1774153487@sss.pgh.pa.us>
 <CAD5tBcLVWKnph3iB-VPuPKR0dCckOJRFZW2-4H7HTTmhw8-vOg@mail.gmail.com>
 <2790913.1774200584@sss.pgh.pa.us> <2880042.1774203473@sss.pgh!!.pa.us>
 <CAD5tBcLsYDz+Nzx8MryjxiKaN3fGKd4ZgXuN1Jn=CYxw9dh+AA@mail.gmail.com>
 <3341199.1774221191@sss.pgh.pa.us> <3424809.1774234940@sss.pgh.pa.us>
 <acIA8KKBTv-T53YG@paquier.xyz>
 <x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
 <CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
 <1624716.1774736283@sss.pgh.pa.us> <1626907.1774737417@sss.pgh.pa.us>
 <97a382c0-1f19-4ea0-951f-e37e6abc34a3@vondra.me>
 <1630755.1774739531@sss.pgh.pa.us>
 <c3a2b730-9d90-4a66-ae5e-50f6d77922d6@vondra.me>
 <CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
 <1873141.1774823011@sss.pgh.pa.us>
In-Reply-To: <1873141.1774823011@sss.pgh.pa.us>
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 1 Apr 2026 15:05:53 +1300
Message-ID: 
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>,
	Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>,
 Amul Sul <sulamul@gmail.com>,
	Zsolt Parragi <zsolt.parragi@percona.com>,
 Robert Haas <robertmhaas@gmail.com>,
	Chao Li <li.evan.chao@gmail.com>,
	Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>,
 Fujii Masao <masao.fujii@gmail.com>,
	Jakub Wartak <jakub.wartak@enterprisedb.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/mixed; boundary="000000000000594a82064e5c8889"
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BhUKG%2BPqz5%3DYQG_%3D8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w%40mail.gmail.com>
Precedence: bulk

--000000000000594a82064e5c8889
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Mar 30, 2026 at 11:23=E2=80=AFAM Tom Lane <tgl@sss.pgh.pa.us> wrote=
:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
> > unlikely to hit this in the wild, and the symptom is a confusing error
> > in a maintenance tool, not corruption, so I don't think this is a big
> > deal.  I might still try teaching the astreamer code to understand PAX
> > 1.0 when it sees it in the next cycle though, for the benefit of
> > FreeBSD users.
>
> I agree that this isn't too critical if the effects are confined to
> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
> parse files made by tar(1) and not by our own sparseness-ignorant
> tar-writing code.  If they can be, that'd be a higher-priority reason
> to fill in this gap.

I pushed the workaround for the test.

Yeah I can't see any reason why pg_verifybackup --wal-path=3Dfoo.tar
won't suffer the same problem in the wild.  Again, it's not the end of
the world because it'll just fail and you'll probably eventually
figure out why.  So perhaps we should just improve our detection of
archives that we can't handle?  Straw man algorithm:

If you can't find $NAME in the archive, then check if PaxHeaders/$NAME
exists, and if so, fail with 'unsupported TAR format for WAL file "%s"
in archive "%s"' instead.  That'd probably work well enough in
practice, because astreamer_tar.c treats PAX extended header
pseudo-files as regular files (they're not, they have type 'x'), and
both GNU and BSD tar happen to use that.

POSIX doesn't require that naming, so it would in theory be more
correct to teach astreamer_tar.c to recognise PAX extended headers and
fish out enough information and link it to the following archive
member, but a simple test to improve error messaging seems like the
right level of effort here.

Here's a test patch that shows the problem on any system with GNU tar
or BSD tar and a file system that supports sparse files.  The test
succeeds because it looks for "error: could not find WAL" but the idea
would be to change it to look for a new error message like that.  My
motivation was to make this reproducible on any system, in case that's
helpful for Amul and Andrew if they're interested in trying to improve
this edge case in time for the release.  Otherwise I'll come back to
it, but probably not in time...

--000000000000594a82064e5c8889
Content-Type: text/x-patch; charset="US-ASCII";
	name="0001-Add-a-pg_waldump-test-with-GNU-tar-PAX-format.patch"
Content-Disposition: attachment;
	filename="0001-Add-a-pg_waldump-test-with-GNU-tar-PAX-format.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mnfdyx5p0>
X-Attachment-Id: f_mnfdyx5p0

RnJvbSAwODRkNzFmODExNDNmMDQ2MmNhZjAzNTY5NzIyYjVmMGIyYTE0N2U2IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgTXVucm8gPHRob21hcy5tdW5yb0BnbWFpbC5jb20+
CkRhdGU6IE1vbiwgMzAgTWFyIDIwMjYgMTg6MjA6MDkgKzEzMDAKU3ViamVjdDogW1BBVENIXSBB
ZGQgYSBwZ193YWxkdW1wIHRlc3Qgd2l0aCBHTlUgdGFyIFBBWCBmb3JtYXQuCgpYWFggVXBkYXRl
IHRoaXMgdG8gdGVzdCBmb3IgYSBuZXcgaW1wcm92ZWQgZXJyb3IgbWVzc2FnZSEKClhYWCBTaG91
bGQgdGhpcyB0ZXN0IHJ1biBmb3IgYWxsIHRoZSBzY2VuYXJpb3M/ICBEb2Vzbid0IHNlZW0gbGlr
ZQpjb21wcmVzc2lvbiBpcyByZWxldmFudCB0byB0aGlzIHByb2JsZW0gc28gSSBqdXN0IGFkZGVk
IGl0IGFzIGEKc3RhbmRhbG9uZSB0ZXN0Li4uCgpYWFggTm8gZG91YnQgdGhlIHBlcmwgaXNuJ3Qg
dGhlIGdyZWF0ZXN0Li4uCi0tLQogc3JjL2Jpbi9wZ193YWxkdW1wL3QvMDAxX2Jhc2ljLnBsIHwg
NzMgKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLQogMSBmaWxlIGNoYW5nZWQsIDY5IGlu
c2VydGlvbnMoKyksIDQgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvc3JjL2Jpbi9wZ193YWxk
dW1wL3QvMDAxX2Jhc2ljLnBsIGIvc3JjL2Jpbi9wZ193YWxkdW1wL3QvMDAxX2Jhc2ljLnBsCmlu
ZGV4IGNlMWY2YWEzMGMwLi43ZjhhMzE5Yzg1ZCAxMDA2NDQKLS0tIGEvc3JjL2Jpbi9wZ193YWxk
dW1wL3QvMDAxX2Jhc2ljLnBsCisrKyBiL3NyYy9iaW4vcGdfd2FsZHVtcC90LzAwMV9iYXNpYy5w
bApAQCAtNiw2ICs2LDcgQEAgdXNlIHdhcm5pbmdzIEZBVEFMID0+ICdhbGwnOwogdXNlIEN3ZDsK
IHVzZSBGaWxlOjpDb3B5OwogdXNlIFBvc3RncmVTUUw6OlRlc3Q6OkNsdXN0ZXI7Cit1c2UgUG9z
dGdyZVNRTDo6VGVzdDo6UmVjdXJzaXZlQ29weTsKIHVzZSBQb3N0Z3JlU1FMOjpUZXN0OjpVdGls
czsKIHVzZSBUZXN0OjpNb3JlOwogdXNlIExpc3Q6OlV0aWwgcXcoc2h1ZmZsZSk7CkBAIC0yMTIs
OSArMjEzLDEzIEBAICRub2RlLT5zYWZlX3BzcWwoJ3Bvc3RncmVzJywKIAlxcXtTRUxFQ1QgcGdf
bG9naWNhbF9lbWl0X21lc3NhZ2UodHJ1ZSwgJ3Rlc3QgMDI2JywgcmVwZWF0KCd4eXp4eicsIDEy
MzQ1NikpfQogKTsKIAotbXkgKCRlbmRfbHNuLCAkZW5kX3dhbGZpbGUpID0gc3BsaXQgL1x8LywK
K215ICgkZW5kX2xzbiwgJGVuZF93YWxmaWxlLCAkd2FsX3NlZ3NpemUpID0gc3BsaXQgL1x8LywK
ICAgJG5vZGUtPnNhZmVfcHNxbCgncG9zdGdyZXMnLAotCXF7U0VMRUNUIHBnX2N1cnJlbnRfd2Fs
X2luc2VydF9sc24oKSwgcGdfd2FsZmlsZV9uYW1lKHBnX2N1cnJlbnRfd2FsX2luc2VydF9sc24o
KSl9CisJcXtTRUxFQ1QgcGdfY3VycmVudF93YWxfaW5zZXJ0X2xzbigpLAorCQkJIHBnX3dhbGZp
bGVfbmFtZShwZ19jdXJyZW50X3dhbF9pbnNlcnRfbHNuKCkpLAorCQkJIHNldHRpbmcKKyAgICAg
ICAgRlJPTSBwZ19zZXR0aW5ncworICAgICAgIFdIRVJFIG5hbWUgPSAnd2FsX3NlZ21lbnRfc2l6
ZSd9CiAgICk7CiAKIG15ICRkZWZhdWx0X3RzX29pZCA9ICRub2RlLT5zYWZlX3BzcWwoJ3Bvc3Rn
cmVzJywKQEAgLTMzOSw3ICszNDQsNyBAQCBzdWIgdGVzdF9wZ193YWxkdW1wCiAjIENyZWF0ZSBh
IHRhciBhcmNoaXZlLCBzaHVmZmxlIHRoZSBmaWxlIG9yZGVyCiBzdWIgZ2VuZXJhdGVfYXJjaGl2
ZQogewotCW15ICgkYXJjaGl2ZSwgJGRpcmVjdG9yeSwgJGNvbXByZXNzaW9uX2ZsYWdzKSA9IEBf
OworCW15ICgkYXJjaGl2ZSwgJGRpcmVjdG9yeSwgJGNvbXByZXNzaW9uX2ZsYWdzLCBAZXh0cmFf
ZmxhZ3MpID0gQF87CiAKIAlteSBAZmlsZXM7CiAJb3BlbmRpciBteSAkZGgsICRkaXJlY3Rvcnkg
b3IgZGllICJvcGVuZGlyOiAkISI7CkBAIC0zNTAsMTIgKzM1NSwxNyBAQCBzdWIgZ2VuZXJhdGVf
YXJjaGl2ZQogCX0KIAljbG9zZWRpciAkZGg7CiAKKwlpZiAoIUBleHRyYV9mbGFncykKKwl7CisJ
CUBleHRyYV9mbGFncyA9IEB0YXJfY19mbGFnczsKKwl9CisKIAlAZmlsZXMgPSBzaHVmZmxlIEBm
aWxlczsKIAogCSMgbW92ZSBpbnRvIHRoZSBXQUwgZGlyZWN0b3J5IGJlZm9yZSBhcmNoaXZpbmcg
ZmlsZXMKIAlteSAkY3dkID0gZ2V0Y3dkOwogCWNoZGlyKCRkaXJlY3RvcnkpIHx8IGRpZSAiY2hk
aXI6ICQhIjsKLQljb21tYW5kX29rKFskdGFyLCBAdGFyX2NfZmxhZ3MsICRjb21wcmVzc2lvbl9m
bGFncywgJGFyY2hpdmUsIEBmaWxlc10pOworCWNvbW1hbmRfb2soWyR0YXIsIEBleHRyYV9mbGFn
cywgJGNvbXByZXNzaW9uX2ZsYWdzLCAkYXJjaGl2ZSwgQGZpbGVzXSk7CiAJY2hkaXIoJGN3ZCkg
fHwgZGllICJjaGRpcjogJCEiOwogfQogCkBAIC00NzcsNCArNDg3LDU5IEBAIGZvciBteSAkc2Nl
bmFyaW8gKEBzY2VuYXJpb3MpCiAJfQogfQogCitTS0lQOgorCXNraXAgInRhciBjb21tYW5kIGlz
IG5vdCBhdmFpbGFibGUiLCAxCisJCWlmICFkZWZpbmVkICR0YXI7CisKKwlteSBAc3BhcnNlX2Zs
YWdzOworCisJIyBUZWxsICRUQVIgdG8gdXNlIEdOVSB0YXIncyBQQVggc3BhcnNlIGZpbGUgYXJj
aGl2ZSBmb3JtYXQsIHNvIHdlIGNhbiB0ZXN0CisJIyBvdXIgaGFuZGxpbmcgb2YgdGhhdC4KKwor
CSMgR05VIHRhcgorCUBzcGFyc2VfZmxhZ3MgPSAoIi0tc3BhcnNlIiwgIi0tZm9ybWF0PXBheCIp
CisJCWlmIHN5c3RlbSgiJHRhciAtLXNwYXJzZSAtLWZvcm1hdD1wYXggLWMgIiAuCisJCQkJICAk
bm9kZS0+ZGF0YV9kaXIgLiAiL3BnX3dhbC8qIC9kZXYvbnVsbCA+IC9kZXYvbnVsbCIpID09IDA7
CisJIyBCU0QgdGFyICh0aGlzIGlzIHRoZSBkZWZhdWx0LCBidXQgd2Ugc3RpbGwgbmVlZCB0byBk
ZXRlY3QgQlNEIHRhcikKKwlAc3BhcnNlX2ZsYWdzID0gKCItLXJlYWQtc3BhcnNlIiwgIi0tZm9y
bWF0PXBheCIpCisJCWlmIHN5c3RlbSgiJHRhciAtLXJlYWQtc3BhcnNlIC0tZm9ybWF0PXBheCAt
YyAiIC4KKwkJCQkgICRub2RlLT5kYXRhX2RpciAuICIvcGdfd2FsLyogL2Rldi9udWxsID4gL2Rl
di9udWxsIikgPT0gMDsKKworCXNraXAgInRhciBjb21tYW5kIGRvZXNuJ3Qgc3VwcG9ydCBHTlUg
UEFYIGZvcm1hdCBmb3Igc3BhcnNlIGZpbGVzIiwgMQorCQlpZiAhQHNwYXJzZV9mbGFnczsKKwor
CVBvc3RncmVTUUw6OlRlc3Q6OlJlY3Vyc2l2ZUNvcHk6OmNvcHlwYXRoKCRub2RlLT5kYXRhX2Rp
ciAuICcvcGdfd2FsJywKKwkJCQkJCQkJCQkJICAkdG1wX2RpciAuICcvcGdfd2FsX3NwYXJzZScp
OworCisJIyB0cnVuY2F0ZSB0aGUgdW51c2VkIHBhcnQgb2YgZmluYWwgV0FMIGZpbGUKKwlteSAk
ZW5kX2J5dGUgPSAkZW5kX2xzbjsKKwkkZW5kX2J5dGUgPX4gcy9cLy8vOworCSRlbmRfYnl0ZSA9
IGhleCgkZW5kX2J5dGUpOworCSRlbmRfYnl0ZSAlPSAkd2FsX3NlZ3NpemU7CisJdHJ1bmNhdGUg
JHRtcF9kaXIgLiAnL3BnX3dhbF9zcGFyc2UvJyAuICRlbmRfd2FsZmlsZSwgJGVuZF9ieXRlOwor
CisJIyBub3cgcmUtZXh0ZW5kIGl0IHRvIGNyZWF0ZSBhIGhvbGUKKwl0cnVuY2F0ZSAkdG1wX2Rp
ciAuICcvcGdfd2FsX3NwYXJzZS8nIC4gJGVuZF93YWxmaWxlLCAkd2FsX3NlZ3NpemU7CisKKwkj
IFhYWCBtYXliZSB3ZSBzaG91bGQgZGV0ZWN0IHNwYXJzZSBmaWxlcyB3aXRoIHN0YXQgKHNpemUg
PiBibG9ja3MgKiBibG9jaworCSMgc2l6ZT8pLCBhbmQgc2tpcCB0aGUgdGVzdCBpZiB0cnVuY2F0
ZSBmYWlsZWQgdG8gbWFrZSBvbmUuLi4gdGhhdAorCSMgbWlnaHQgaGFwcGVuIG9uIGVnIHdpbmRv
d3MgSSB0aGluaz8gIG90aGVyd2lzZSB3ZSdkIGhhdmUgdG8gdG9sZXJhdGUKKwkjIHRoZSBwZ193
YWxkdW1wIGNvbW1hbmQgc3VjY2VlZGluZyBPUiBmYWlsaW5nIHdpdGggYSBjZXJ0YWluIG1lc3Nh
Z2UKKworCWdlbmVyYXRlX2FyY2hpdmUoJHRtcF9kaXIgLiAnL3BnX3dhbF9zcGFyc2UudGFyJywK
KwkJCQkJICR0bXBfZGlyIC4gJy9wZ193YWxfc3BhcnNlJywKKwkJCQkJICctY2YnLAorCQkJCQkg
QHNwYXJzZV9mbGFncyk7CisKKwkjIFhYWCBjaGFuZ2UgdGhpcyB0byBjaGVjayBmb3IgbmV3IGlt
cHJvdmVkIGVycm9yIG1lc3NhZ2UKKwljb21tYW5kX2ZhaWxzX2xpa2UoCisJCVsKKwkJCSdwZ193
YWxkdW1wJywKKwkJCSctLXBhdGgnID0+ICR0bXBfZGlyIC4gJy9wZ193YWxfc3BhcnNlLnRhcics
CisJCQknLS1zdGFydCcgPT4gJHN0YXJ0X2xzbiwKKwkJCSctLWVuZCcgPT4gJGVuZF9sc24sCisJ
CV0sCisJCXFyL2Vycm9yOiBjb3VsZCBub3QgZmluZCBXQUwvLAorCQknZmFpbHMgd2l0aCBHTlUg
dGFyIFBBWC1mb3JtYXQgc3BhcnNlIGZpbGVzJyk7CisKIGRvbmVfdGVzdGluZygpOwotLSAKMi41
My4wCgo=
--000000000000594a82064e5c8889--