Content-Type: multipart/alternative;
 boundary="------------T0dxcgrx2JiXMZF6HewjA4Vs"
Message-ID: <94547580-4c88-4e3b-ab53-f501a2da6f06@dunslane.net>
Date: Thu, 2 Apr 2026 07:36:25 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>,
 Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>,
 Zsolt Parragi <zsolt.parragi@percona.com>,
 Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>,
 Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>,
 Fujii Masao <masao.fujii@gmail.com>,
 Jakub Wartak <jakub.wartak@enterprisedb.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <2250061.1774104346@sss.pgh.pa.us>
 <CAD5tBcLVWKnph3iB-VPuPKR0dCckOJRFZW2-4H7HTTmhw8-vOg@mail.gmail.com>
 <2790913.1774200584@sss.pgh.pa.us> <2880042.1774203473@sss.pgh!!.pa.us>
 <CAD5tBcLsYDz+Nzx8MryjxiKaN3fGKd4ZgXuN1Jn=CYxw9dh+AA@mail.gmail.com>
 <3341199.1774221191@sss.pgh.pa.us> <3424809.1774234940@sss.pgh.pa.us>
 <acIA8KKBTv-T53YG@paquier.xyz>
 <x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
 <CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
 <1624716.1774736283@sss.pgh.pa.us> <1626907.1774737417@sss.pgh.pa.us>
 <97a382c0-1f19-4ea0-951f-e37e6abc34a3@vondra.me>
 <1630755.1774739531@sss.pgh.pa.us>
 <c3a2b730-9d90-4a66-ae5e-50f6d77922d6@vondra.me>
 <CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
 <1873141.1774823011@sss.pgh.pa.us>
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
 <3049460.1775067940@sss.pgh.pa.us>
 <CA+hUKGJv2hpjMDwG31JSR7TfFUxNPPtQKMm6mHVVRMAT0XppBQ@mail.gmail.com>
From: Andrew Dunstan <andrew@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAlwQTAQgAQQIbAwIX
 gAIZAQULCQgHAwUVCgkICwUWAgMBAAIeBRYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNGd
 BQkdEO8nAAoJEJn6f8tZ/DuBq74H/jkTR4Zi3stbw+xC7v2u3QozssK7MYPL2AsVfh7OealS
 h182fiWXpfvmmAB7WUHbhk9GC2RAOnHI/2d2jgKaMLAHsGYOT0YopTVIwRY43fCw/mK67yxc
 wmDcX+zyKfLaivNbf5A7QPLNwda98bEAMSJ8Sn652Uc6cA8t3uKGsVzbRBQOoYzjgvBCfSrE
 9ql3PDNg0l4BfAqabd2f70ZUm9VAMEPrgv/v2xI7M2XiL4g5BVmqLCOwxLM8RMCotCuoweUr
 VO43DeBCIDwLxotMJKvGWDjBzQYlU1NPUAtNcz/gN9ITUe1VUGjyvGj4u1lxBOcQQUw7l1+T
 5moZ4iZxXzvOwE0ETspYWQEIANGc4zQULOxhbqO2dyD51YhqCNRmm9oKWaqf+wmW4tpDe/VV
 cxAnNizd4LWCHfzpb5cHAtGkOPePMfzWVf6nvdF7d3eglbtf59+zG7O7llV0xSSoFiieQBsr
 GvqDInXYX/4mRRXMtyhM353/tixC9RWLs1oofyYmCPPXXY7h9R7en3B8BoVrRFcdzlIY/NFN
 hFGW/9dkEiGjgna2Rk6e15kln4ZvFBWUg23p93w/pqXcxY6+k/8TEk+C4R+M6w7o2PLGOjdZ
 +kPiUcw5H85zf/yZJwQXzisXaNduwWB6Vads9YC9dj6kPR1c4VGRqAaYL++LAEOqrlvm2Tvq
 QqZRtnEAEQEAAcLAfAQYAQgAJgIbDBYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNI2BQkd
 EODdAAoJEJn6f8tZ/DuBfw0IAKTsfD40teP/pp+bsLLMSxPXUYrrprTj7WFB5v61p6dkpSr/
 qXmMlyahdxQFaPmfVgVirB1Vk/kHiWNnnGjfUV9nB2Zg9LI0Xb9/ts3LsUiRWXzG3tkMY6XL
 vsVOxW4XFRND9l2q+WW93aZ1DZl+fqWfYgMvsusFRhmGFOKTRfKPta2Pkv+AhA24N4+PrR5p
 bU4k2MO8PAGiK8eaYKGFG1bHKuAvoDoF7WXJ3FHxuWqLnKEt4dfOLm5pAe3zq1Lt6q8azT9i
 QWGpSAK5vQUWQHBHpiDjdPeqKZ6HiAXIIKfSmb+jrvXBqoP+D6/K7rUjG2aXiRtTIAXms9sm
 VRu7cmw=
In-Reply-To: 
 <CA+hUKGJv2hpjMDwG31JSR7TfFUxNPPtQKMm6mHVVRMAT0XppBQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/94547580-4c88-4e3b-ab53-f501a2da6f06%40dunslane.net>
Precedence: bulk

This is a multi-part message in MIME format.
--------------T0dxcgrx2JiXMZF6HewjA4Vs
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 2026-04-01 We 8:16 PM, Thomas Munro wrote:
> On Thu, Apr 2, 2026 at 7:25 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> Also, if we are admitting the possibility that what we are reading
>> was made by a platform-supplied tar and not our own code, I think
>> it verges on lunacy to behave as though unsupported typeflags are
>> regular files.
> Yeah, if this is the first time we parse files we didn't make then
> that makes total sense.  I was a bit unsure of that question when I
> suggested we reject pax only after we've failed to find a file, in
> case there are scenarios that work today with harmless ignorable pax
> headers that don't change the file name.
>
>> So I think we need something more or less like the attached.
> LGTM.  Tested with both tars here.  I updated that little test patch
> for this.  Not sure if you think it's worth a test though, now that
> it's so simple.
>
> @Andrew: I tried usingFile::Spec->devnull() this time.  Are you able
> to check if this works OK on Windows, applied on top of Tom's patch?
> AFAIK should be able to run this new test and pass, not skip it.  But
> it could be that the shell invocation needs tweaking.  It's hard to
> tell from CI.  (Huh, apparently Windows ships a copy of BSD tar as
> C:\Windows\System32\tar.exe these days.)


Yes, that appears to work. I would put a "2>&1" at the end - we don't 
care about the output, just whether or not it succeeds:


C:\Windows\system32>perl -MFile::Spec -e "print File::Spec->devnull();"
nul

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 && echo hello

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 || echo hello
hello


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

--------------T0dxcgrx2JiXMZF6HewjA4Vs
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 2026-04-01 We 8:16 PM, Thomas Munro
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CA+hUKGJv2hpjMDwG31JSR7TfFUxNPPtQKMm6mHVVRMAT0XppBQ@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">On Thu, Apr 2, 2026 at 7:25 AM Tom Lane <a class="moz-txt-link-rfc2396E" href="mailto:tgl@sss.pgh.pa.us">&lt;tgl@sss.pgh.pa.us&gt;</a> wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">Also, if we are admitting the possibility that what we are reading
was made by a platform-supplied tar and not our own code, I think
it verges on lunacy to behave as though unsupported typeflags are
regular files.
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
Yeah, if this is the first time we parse files we didn't make then
that makes total sense.  I was a bit unsure of that question when I
suggested we reject pax only after we've failed to find a file, in
case there are scenarios that work today with harmless ignorable pax
headers that don't change the file name.

</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">So I think we need something more or less like the attached.
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
LGTM.  Tested with both tars here.  I updated that little test patch
for this.  Not sure if you think it's worth a test though, now that
it's so simple.

@Andrew: I tried using <a class="moz-txt-link-freetext" href="File::Spec">File::Spec</a>-&gt;devnull() this time.  Are you able
to check if this works OK on Windows, applied on top of Tom's patch?
AFAIK should be able to run this new test and pass, not skip it.  But
it could be that the shell invocation needs tweaking.  It's hard to
tell from CI.  (Huh, apparently Windows ships a copy of BSD tar as
C:\Windows\System32\tar.exe these days.)</pre>
    </blockquote>
    <p><br>
    </p>
    <p>Yes, that appears to work. I would put a "2&gt;&amp;1" at the end
      - we don't care about the output, just whether or not it succeeds:</p>
    <p><br>
    </p>
    <p>C:\Windows\system32&gt;perl -MFile::Spec -e "print
      <a class="moz-txt-link-freetext" href="File::Spec">File::Spec</a>-&gt;devnull();"<br>
      nul<br>
    </p>
    <p>C:\Windows\system32&gt;tar --no-read-sparse -c - nul &gt; nul
      2&gt;&amp;1 &amp;&amp; echo hello<br>
      <br>
      C:\Windows\system32&gt;tar --no-read-sparse -c - nul &gt; nul
      2&gt;&amp;1 || echo hello<br>
      hello<br>
    </p>
    <p><br>
    </p>
    <p>cheers</p>
    <p><br>
    </p>
    <p>andrew<br>
    </p>
    <p><span style="white-space: pre-wrap">
</span></p>
    <pre class="moz-signature"
    signature-switch-id="d0437855-2267-4610-80b3-83167ec45b0b" cols="72">--
Andrew Dunstan
EDB: <a class="moz-txt-link-freetext" href="https://www.enterprisedb.com">https://www.enterprisedb.com</a>
</pre>
  </body>
</html>

--------------T0dxcgrx2JiXMZF6HewjA4Vs--