MIME-Version: 1.0
References: <CAAJ_b94bqdWN3h2J-PzzzQ2Npbwct5ZQHggn_QoYGhC2rn-=WQ@mail.gmail.com>
 <CAAJ_b97PQjE4kFD8Qk6UvtLrfPMixw1nxBz0OP5Z2WB2B-uMxQ@mail.gmail.com>
 <CAAJ_b97JA8ehy_UDddrnGwDt9HG5NmJq8ATtmeMqo7YD-=tLyQ@mail.gmail.com>
 <CA+TgmoZjhWDG_AR1i+L1yss-wbuWvxrdRwSdVUUUnVPrJV2CnQ@mail.gmail.com>
 <CAAJ_b94Uh+b41LQG45bZFK+i62EVvv972LiGWWWuR64=-64rTQ@mail.gmail.com>
 <CA+TgmobF5c7ZcZHdEhqwNxGDZzWG2bDtpRaDtoVELWX_VHs_1A@mail.gmail.com>
 <CAAJ_b94gK1np8d1h-2c1YoCccGXr4zspTa-FC7X_bfXZNz=-DA@mail.gmail.com>
 <CA+TgmoayDY5b+bP1vRRN7A3xOP-=+tK13B2C1g-Xm1j4WTrT9Q@mail.gmail.com>
 <CAAJ_b97JAF+Zuoh2FBO79hVwLeaBPwsbXw-fY+313a7LfRQ-Bg@mail.gmail.com>
 <CA+Tgmoardk4VuthHc23vov+AVkhq7eT0mFUs-2ctAnP1uiTaog@mail.gmail.com>
 <CAAJ_b959x5VjmLJFmN78r_QohQuuj=fde11VbbAOHn5TzgEzng@mail.gmail.com>
 <CAAJ_b97_N+0sipFyq80n0jX-nKcjcQEMOSTVg8DsqkHR8dW_Sw@mail.gmail.com>
 <CAKZiRmyDk5KqovS9Ez3iFHd+p-TChSt2QTtWkwJ5Ya-+4gg21g@mail.gmail.com>
 <CAAJ_b956a+e8-HNEpeJ60ByFv7XJRqECPu3B0dozv0ChMRTPbQ@mail.gmail.com>
 <CAAJ_b94f6sUDWiZY90O-t7SWWeSK0nMWT7AsydKkpAL90m0oMQ@mail.gmail.com>
 <731ADE6F-01C5-4996-BAEE-5851DFC3F502@gmail.com> <CAAJ_b94SEcBVJcsp0Y1-YvLqZWBHPQH4FhRzSJfaH_ah_eL_FQ@mail.gmail.com>
 <CAAJ_b97nn9denP2SAjuXyjwbd3is-VnZVSkiRMJ-5YNiKfO9MA@mail.gmail.com>
 <CA+Tgmob_DB9QHDOxnP7a5Y0yJdeGqY8YNi+uK_811y7cN4mxYA@mail.gmail.com>
 <CAAJ_b97VUiP-DbLNe-ddq64J_RiB4ZcPgAjHkJH-0dbzgjR++A@mail.gmail.com>
 <CA+TgmoYMtcZBaqy9r59eDapaDy3WOdepkFFURu9MV-x-kxEbKg@mail.gmail.com> <CAAJ_b95FOeW38gw-3BLmpdnTWHFimopTvf=eTObYUbTOC0x8qg@mail.gmail.com>
In-Reply-To: <CAAJ_b95FOeW38gw-3BLmpdnTWHFimopTvf=eTObYUbTOC0x8qg@mail.gmail.com>
From: Robert Haas <robertmhaas@gmail.com>
Date: Mon, 26 Jan 2026 12:22:33 -0500
Message-ID: <CA+Tgmob=3POOO8st-v-fCjKCKREQ=+gs5_PBQnoFeNBdERfuEg@mail.gmail.com>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
To: Amul Sul <sulamul@gmail.com>
Cc: Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CA%2BTgmob%3D3POOO8st-v-fCjKCKREQ%3D%2Bgs5_PBQnoFeNBdERfuEg%40mail.gmail.com>
Precedence: bulk

On Fri, Jan 23, 2026 at 7:27=E2=80=AFAM Amul Sul <sulamul@gmail.com> wrote:
> Another option I previously considered was adding the filtration logic
> inside the archive streamer itself. However, since the very first read
> is required to calculate the WAL segment size, the filter check cannot
> be performed immediately. However, we could send a signal to the
> archive streamer via privateInfo (e.g., a read_any_wal or
> skip_wal_check boolean flag) to disable the filtration check until the
> size is calculated. But that approach isn't very elegant; if the first
> WAL page we read belongs to a segment we actually want to skip, we
> would still have to run the filter check and handle the skip/removal
> logic outside of the streamer (i.e., inside init_archive_reader()).
> This would result in performing the same filtration check in two
> different places.

I mean, I don't really buy this logic. If the information added to
privateInfo is "here's the LSN before which you can remove stuff," and
it starts out initialized to 0/0, then the read of the first WAL page
causes no problem at all, because nothing is before 0/0. After it gets
updated to some non-zero value, the next call to
astreamer_waldump_content() can handle discarding any data we don't
need.

IMHO, the best argument for keeping things are is that in some cases,
that decision might result in a bit of delay in discarding old data,
but I don't think that really matters. I think all that we care about
is the peak memory utilization of an operation, and I don't think that
an explicit signaling system should increase that at all.

That said, I'm certainly willing to consider other ideas about how
this can work. However, I feel strongly that the logic needs to be not
only correct, but clear and well-explained. Setting cur_wal to NULL to
make the astreamer skip without adequate comments doesn't meet that
standard. Maybe with some better comments it's all right, but frankly
I'm a bit skeptical. Right now, you're using whether or not cur_wal is
NULL as a signal to skip data or not skip data. How is that better
than passing down the LSN and TLI that you want to read next and
letting the astreamer figure out what to do itself? It's a signaling
mechanism either way, but it seems a lot easier to figure out whether
we always keep the LSN and TLI updated properly than to figure out
whether cur_wal is always NULL at exactly the right times.

--=20
Robert Haas
EDB: http://www.enterprisedb.com