MIME-Version: 1.0
References: 
 <CAAJ_b94bqdWN3h2J-PzzzQ2Npbwct5ZQHggn_QoYGhC2rn-=WQ@mail.gmail.com>
 <CAAJ_b97PQjE4kFD8Qk6UvtLrfPMixw1nxBz0OP5Z2WB2B-uMxQ@mail.gmail.com>
 <CAAJ_b97JA8ehy_UDddrnGwDt9HG5NmJq8ATtmeMqo7YD-=tLyQ@mail.gmail.com>
 <CA+TgmoZjhWDG_AR1i+L1yss-wbuWvxrdRwSdVUUUnVPrJV2CnQ@mail.gmail.com>
 <CAAJ_b94Uh+b41LQG45bZFK+i62EVvv972LiGWWWuR64=-64rTQ@mail.gmail.com>
 <CA+TgmobF5c7ZcZHdEhqwNxGDZzWG2bDtpRaDtoVELWX_VHs_1A@mail.gmail.com>
 <CAAJ_b94gK1np8d1h-2c1YoCccGXr4zspTa-FC7X_bfXZNz=-DA@mail.gmail.com>
 <CA+TgmoayDY5b+bP1vRRN7A3xOP-=+tK13B2C1g-Xm1j4WTrT9Q@mail.gmail.com>
 <CAAJ_b97JAF+Zuoh2FBO79hVwLeaBPwsbXw-fY+313a7LfRQ-Bg@mail.gmail.com>
 <CA+Tgmoardk4VuthHc23vov+AVkhq7eT0mFUs-2ctAnP1uiTaog@mail.gmail.com>
 <CAAJ_b959x5VjmLJFmN78r_QohQuuj=fde11VbbAOHn5TzgEzng@mail.gmail.com>
 <CAAJ_b97_N+0sipFyq80n0jX-nKcjcQEMOSTVg8DsqkHR8dW_Sw@mail.gmail.com>
In-Reply-To: 
 <CAAJ_b97_N+0sipFyq80n0jX-nKcjcQEMOSTVg8DsqkHR8dW_Sw@mail.gmail.com>
From: Jakub Wartak <jakub.wartak@enterprisedb.com>
Date: Wed, 19 Nov 2025 09:20:14 +0100
Message-ID: 
 <CAKZiRmyDk5KqovS9Ez3iFHd+p-TChSt2QTtWkwJ5Ya-+4gg21g@mail.gmail.com>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Amul Sul <sulamul@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAKZiRmyDk5KqovS9Ez3iFHd%2Bp-TChSt2QTtWkwJ5Ya-%2B4gg21g%40mail.gmail.com>
Precedence: bulk

On Mon, Nov 17, 2025 at 5:51=E2=80=AFAM Amul Sul <sulamul@gmail.com> wrote:
>
> On Thu, Nov 6, 2025 at 2:33=E2=80=AFPM Amul Sul <sulamul@gmail.com> wrote=
:
> >
> > On Mon, Oct 20, 2025 at 8:05=E2=80=AFPM Robert Haas <robertmhaas@gmail.=
com> wrote:
> > >
> > > On Thu, Oct 16, 2025 at 7:49=E2=80=AFAM Amul Sul <sulamul@gmail.com> =
wrote:
> > > [....]
> > Kindly have a look at the attached version. Thank you !
> >
>
> Attached is the rebased version against the latest master head (e76defbcf=
09).

Hi Amul, thanks for working on this. I haven't really looked at the
source code deeply (I trust Robert eyes much more than mine on this
one), just skimmed a little bit:

1. As stated earlier, get_tmp_walseg_path() is still vulnerable (it
uses predictable path that could be used by attacker in $TMPDIR)

2. On the usability front:

a. If you do `pg_waldump --path pg_wal.tar -s 0/31000000` it will dump
a lot of WAL records and then print final:
pg_waldump: error: could not find file "000000010000000000000034" in archiv=
e

However, with `pg_waldump --path pg_wal.tar -s 0/31000000
--stats=3Drecord` (not passing '-e') it will simply bailout without
printing stats and with error:
pg_waldump: error: could not find file "000000010000000000000034" in archiv=
e

IMHO, it could print stats if it was capable of getting at least 1 WAL reco=
rd.

3. The most critical issue for me was the initial lack of error
pass-through from pg_waldump (when used with WALs in tar) to the
pg_verifybackup. Now it works fine, so thanks for this:

a. pg_waldump is capable of discovering missing WALs as requested and
throwing proper return code (good)
$ /usr/pgsql19/bin/pg_waldump --path pg_wal.tar -s 0/31005F70 -e 0/343D2650=
 -q
pg_waldump: error: could not find file "000000010000000000000034" in archiv=
e
$ echo $?
1
$

b. pg_verifybackup now also complains properly with missing WAL inside tar

$ tar --delete -f pg_wal.tar 000000010000000000000032 # simulate loss of fi=
le
$ tar -tf pg_wal.tar
000000010000000000000031
archive_status/000000010000000000000031.done
archive_status/000000010000000000000032.done
000000010000000000000033
$ grep Start-LSN backup_manifest
{ "Timeline": 1, "Start-LSN": "0/31005F70", "End-LSN": "0/333D2650" }
$ /usr/pgsql19/bin/pg_verifybackup -P /tmp/basebackup/
791372/791372 kB (100%) verified
pg_waldump: error: could not find file "000000010000000000000032" in archiv=
e
pg_verifybackup: error: WAL parsing failed for timeline 1
$ echo $?
1
$

-J.