Content-Type: multipart/alternative;
 boundary="------------tb0zVIB5aJ00hphI1maTkeGw"
Message-ID: <7c394909-7680-4305-885e-8c7f18ed4441@dunslane.net>
Date: Wed, 1 Apr 2026 06:39:05 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>,
 Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>,
 Zsolt Parragi <zsolt.parragi@percona.com>,
 Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>,
 Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>,
 Fujii Masao <masao.fujii@gmail.com>,
 Jakub Wartak <jakub.wartak@enterprisedb.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <2250061.1774104346@sss.pgh.pa.us>
 <2555285.1774131847@sss.pgh.pa.us> <2609460.1774153487@sss.pgh.pa.us>
 <CAD5tBcLVWKnph3iB-VPuPKR0dCckOJRFZW2-4H7HTTmhw8-vOg@mail.gmail.com>
 <2790913.1774200584@sss.pgh.pa.us> <2880042.1774203473@sss.pgh!!.pa.us>
 <CAD5tBcLsYDz+Nzx8MryjxiKaN3fGKd4ZgXuN1Jn=CYxw9dh+AA@mail.gmail.com>
 <3341199.1774221191@sss.pgh.pa.us> <3424809.1774234940@sss.pgh.pa.us>
 <acIA8KKBTv-T53YG@paquier.xyz>
 <x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
 <CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
 <1624716.1774736283@sss.pgh.pa.us> <1626907.1774737417@sss.pgh.pa.us>
 <97a382c0-1f19-4ea0-951f-e37e6abc34a3@vondra.me>
 <1630755.1774739531@sss.pgh.pa.us>
 <c3a2b730-9d90-4a66-ae5e-50f6d77922d6@vondra.me>
 <CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
 <1873141.1774823011@sss.pgh.pa.us>
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
From: Andrew Dunstan <andrew@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAlwQTAQgAQQIbAwIX
 gAIZAQULCQgHAwUVCgkICwUWAgMBAAIeBRYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNGd
 BQkdEO8nAAoJEJn6f8tZ/DuBq74H/jkTR4Zi3stbw+xC7v2u3QozssK7MYPL2AsVfh7OealS
 h182fiWXpfvmmAB7WUHbhk9GC2RAOnHI/2d2jgKaMLAHsGYOT0YopTVIwRY43fCw/mK67yxc
 wmDcX+zyKfLaivNbf5A7QPLNwda98bEAMSJ8Sn652Uc6cA8t3uKGsVzbRBQOoYzjgvBCfSrE
 9ql3PDNg0l4BfAqabd2f70ZUm9VAMEPrgv/v2xI7M2XiL4g5BVmqLCOwxLM8RMCotCuoweUr
 VO43DeBCIDwLxotMJKvGWDjBzQYlU1NPUAtNcz/gN9ITUe1VUGjyvGj4u1lxBOcQQUw7l1+T
 5moZ4iZxXzvOwE0ETspYWQEIANGc4zQULOxhbqO2dyD51YhqCNRmm9oKWaqf+wmW4tpDe/VV
 cxAnNizd4LWCHfzpb5cHAtGkOPePMfzWVf6nvdF7d3eglbtf59+zG7O7llV0xSSoFiieQBsr
 GvqDInXYX/4mRRXMtyhM353/tixC9RWLs1oofyYmCPPXXY7h9R7en3B8BoVrRFcdzlIY/NFN
 hFGW/9dkEiGjgna2Rk6e15kln4ZvFBWUg23p93w/pqXcxY6+k/8TEk+C4R+M6w7o2PLGOjdZ
 +kPiUcw5H85zf/yZJwQXzisXaNduwWB6Vads9YC9dj6kPR1c4VGRqAaYL++LAEOqrlvm2Tvq
 QqZRtnEAEQEAAcLAfAQYAQgAJgIbDBYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNI2BQkd
 EODdAAoJEJn6f8tZ/DuBfw0IAKTsfD40teP/pp+bsLLMSxPXUYrrprTj7WFB5v61p6dkpSr/
 qXmMlyahdxQFaPmfVgVirB1Vk/kHiWNnnGjfUV9nB2Zg9LI0Xb9/ts3LsUiRWXzG3tkMY6XL
 vsVOxW4XFRND9l2q+WW93aZ1DZl+fqWfYgMvsusFRhmGFOKTRfKPta2Pkv+AhA24N4+PrR5p
 bU4k2MO8PAGiK8eaYKGFG1bHKuAvoDoF7WXJ3FHxuWqLnKEt4dfOLm5pAe3zq1Lt6q8azT9i
 QWGpSAK5vQUWQHBHpiDjdPeqKZ6HiAXIIKfSmb+jrvXBqoP+D6/K7rUjG2aXiRtTIAXms9sm
 VRu7cmw=
In-Reply-To: 
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/7c394909-7680-4305-885e-8c7f18ed4441%40dunslane.net>
Precedence: bulk

This is a multi-part message in MIME format.
--------------tb0zVIB5aJ00hphI1maTkeGw
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
> On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> Thomas Munro<thomas.munro@gmail.com> writes:
>>> Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
>>> unlikely to hit this in the wild, and the symptom is a confusing error
>>> in a maintenance tool, not corruption, so I don't think this is a big
>>> deal.  I might still try teaching the astreamer code to understand PAX
>>> 1.0 when it sees it in the next cycle though, for the benefit of
>>> FreeBSD users.
>> I agree that this isn't too critical if the effects are confined to
>> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
>> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
>> parse files made by tar(1) and not by our own sparseness-ignorant
>> tar-writing code.  If they can be, that'd be a higher-priority reason
>> to fill in this gap.
> I pushed the workaround for the test.


It occurred to me this morning that we probably shouldn't run this test 
on Windows, and if we do we shouldn't be using /dev/null (the Windows 
equivalent of which is just "nul"). The simplest fix would just be to 
add a "!$windows_os" to the if test.


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

--------------tb0zVIB5aJ00hphI1maTkeGw
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 2026-03-31 Tu 10:05 PM, Thomas Munro
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">On Mon, Mar 30, 2026 at 11:23 AM Tom Lane <a class="moz-txt-link-rfc2396E" href="mailto:tgl@sss.pgh.pa.us">&lt;tgl@sss.pgh.pa.us&gt;</a> wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">Thomas Munro <a class="moz-txt-link-rfc2396E" href="mailto:thomas.munro@gmail.com">&lt;thomas.munro@gmail.com&gt;</a> writes:
</pre>
        <blockquote type="cite">
          <pre wrap="" class="moz-quote-pre">Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
unlikely to hit this in the wild, and the symptom is a confusing error
in a maintenance tool, not corruption, so I don't think this is a big
deal.  I might still try teaching the astreamer code to understand PAX
1.0 when it sees it in the next cycle though, for the benefit of
FreeBSD users.
</pre>
        </blockquote>
        <pre wrap="" class="moz-quote-pre">
I agree that this isn't too critical if the effects are confined to
pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
astreamer_tar.c, but it's not clear to me if they'd ever be asked to
parse files made by tar(1) and not by our own sparseness-ignorant
tar-writing code.  If they can be, that'd be a higher-priority reason
to fill in this gap.
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
I pushed the workaround for the test.</pre>
    </blockquote>
    <p><br>
    </p>
    <p>It occurred to me this morning that we probably shouldn't run
      this test on Windows, and if we do we shouldn't be using /dev/null
      (the Windows equivalent of which is just "nul"). The simplest fix
      would just be to add a "!$windows_os" to the if test.</p>
    <p><br>
    </p>
    <p>cheers</p>
    <p><br>
    </p>
    <p>andrew<br>
    </p>
    <p><span style="white-space: pre-wrap">
</span></p>
    <pre class="moz-signature"
    signature-switch-id="d0437855-2267-4610-80b3-83167ec45b0b" cols="72">--
Andrew Dunstan
EDB: <a class="moz-txt-link-freetext" href="https://www.enterprisedb.com">https://www.enterprisedb.com</a>
</pre>
  </body>
</html>

--------------tb0zVIB5aJ00hphI1maTkeGw--