MIME-Version: 1.0
References: <2250061.1774104346@sss.pgh.pa.us>
 <3341199.1774221191@sss.pgh.pa.us>
 <3424809.1774234940@sss.pgh.pa.us> <acIA8KKBTv-T53YG@paquier.xyz>
 <x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
 <CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
 <1624716.1774736283@sss.pgh.pa.us> <1626907.1774737417@sss.pgh.pa.us>
 <97a382c0-1f19-4ea0-951f-e37e6abc34a3@vondra.me>
 <1630755.1774739531@sss.pgh.pa.us>
 <c3a2b730-9d90-4a66-ae5e-50f6d77922d6@vondra.me>
 <CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
 <1873141.1774823011@sss.pgh.pa.us>
 <CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
 <3049460.1775067940@sss.pgh.pa.us> <3118179.1775092964@sss.pgh.pa.us>
 <CA+hUKG+-pn14s_tjEBO6YKHmc=uRhGVn=w2oM91KKnEUc7pH0Q@mail.gmail.com>
 <3565835.1775147392@sss.pgh.pa.us> <3579709.1775151816@sss.pgh.pa.us>
 <63de1553-829a-488d-8ee0-976afb8dd32c@vondra.me>
 <3586483.1775155672@sss.pgh.pa.us>
 <3676229.1775170250@sss.pgh.pa.us>
 <CA+hUKGKfti_FMFuduXEZs96W5Boce9gSLZ5Ei158dFiuLuWLgA@mail.gmail.com>
 <3686764.1775175097@sss.pgh.pa.us>
In-Reply-To: <3686764.1775175097@sss.pgh.pa.us>
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 3 Apr 2026 13:47:28 +1300
Message-ID: 
 <CA+hUKGLNFBBszi7XrTxmW3XWdS8Ojz+H5Jh3XES=1U02F4fVgA@mail.gmail.com>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>,
	Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>,
 Amul Sul <sulamul@gmail.com>,
	Zsolt Parragi <zsolt.parragi@percona.com>,
 Robert Haas <robertmhaas@gmail.com>,
	Chao Li <li.evan.chao@gmail.com>,
	Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>,
 Fujii Masao <masao.fujii@gmail.com>,
	Jakub Wartak <jakub.wartak@enterprisedb.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BhUKGLNFBBszi7XrTxmW3XWdS8Ojz%2BH5Jh3XES%3D1U02F4fVgA%40mail.gmail.com>
Precedence: bulk

On Fri, Apr 3, 2026 at 1:11=E2=80=AFPM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> In any case, this is all completely moot if we don't write code to
> de-sparse a sparse entry: we will not be able to validate WAL data
> if the WAL file is missing some pages.  So I see little point in
> having code that tolerates pax headers if it doesn't also do that.

Yeah.  FWIW I spent a few hours hacking on that the other day and
could decode many files, but I now realise that the task was made more
difficult by a problem you fixed: without header validation, small
mistakes resulted in corruption or went bananas.  With that now
addressed, I hope I can get it into shape and propose it for the next
cycle...

For what it's worth, I was just speculating about how one might
reasonably handle unrecognised *non-standard* header names, not the
POSIX-standardised ones which, you're right, we'd probably need to
grok properly.  If we assumed reasonable engineering decisions
following (what I understood to be) the spirit of pax, maybe we could
assume that new non-standard headers either don't affect file contents
and thus could be ignored (think: GNU.windows.permissions=3D...), or do
affect file contents but have measures in place to prevent unknown
encodings from being exposed to unsuspecting software (think:
deathstation.byte=3D9bit).  That's a position we could choose to take,
anyway, in the absence of a crystal ball...  Fortunately there aren't
really many implementations of POSIX left, so it's not like we're
dealing with the Fermi Paradox here...