pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Wrong LSN in WAL decoding error messages
9+ messages / 4 participants
[nested] [flat]

Commits from this thread:
  2026-11-03 30faf13889  pg_waldump: Align WAL read error locations with other tools
  2026-11-03 01b3d24c41 472da35ac0 092da6511b 5f472ab564 bcb9d90519 3ec9835352  Fix incorrect LSNs in WAL decoding error messages
* Wrong LSN in WAL decoding error messages
@ 2026-10-01 07:17 rahul@rhyadav.dev
  2026-10-07 22:00 ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  0 siblings, 2 replies; 9+ messages in thread

From: rahul@rhyadav.dev @ 2026-10-01 07:17 UTC (permalink / raw)
  To: Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>


Hi,

DecodeXLogRecord() is passed the LSN of the record it decodes, but its
error messages print state->ReadRecPtr instead.  Before the circular
WAL decoding buffer went in (3f1ce97346, v15), the two were the same.
Now records are decoded ahead of the one the reader last returned, so
these messages name an earlier record.  0001 makes them use the lsn
argument.

I noticed this while testing Yuriy's fix for bug #19599 [1], and he
suggested sending it separately.  These checks only see records that
passed the CRC check, so they fire when some code writes a malformed
record, which is when you most want the right LSN.

To trigger one, I set hole_offset to 0 in a full-page image that has
BKPIMAGE_HAS_HOLE set, and recomputed the record CRC.  For that
record, at 0/01C03DC0, master reports:

  pg_waldump: error in WAL record at 0/01C03D38: BKPIMAGE_HAS_HOLE
    set, but hole offset 0 length 4208 block image length 3984 at
    0/01C03D38

  crash recovery: BKPIMAGE_HAS_HOLE set, but hole offset 0 length
    4208 block image length 3984 at 0/01C03CD8

0/01C03D38 is the record just before it and 0/01C03CD8 is three
records back.  With the patches applied, both report 0/01C03DC0, and
crash recovery still ends at the same place.

0002 is separate because it changes long-standing output.
pg_waldump's "error in WAL record at" prefix also uses ReadRecPtr,
which is the last record read successfully rather than where reading
failed.  At the normal end of WAL that gives:

  error in WAL record at 0/017AC370: invalid record length at
  0/017AC3F8: expected at least 24, got 0

With 0002 both LSNs are 0/017AC3F8.  It uses EndRecPtr, as pg_rewind
does.  v14 behaves the same at the end of WAL, so if 0002 is wanted
at all I'd keep it to master.  I can add a check to pg_waldump's TAP
test that the two LSNs match.

0001 applies to REL_19_STABLE as is.  On 15 to 18 the messages still
use %X/%X, so the context needs a small adjustment.

[1] https://postgr.es/m/cd00a0536c674fac9e5dd9f4e27ae868@localhost.localdomain


Regards,
Rahul

Attachments: download all (2 files, .tar.gz)

v1-0002-pg_waldump-Report-the-position-of-a-WAL-read-erro.patch (application/octet-stream, 1.4K) download From 27363ec05340a56435843345ac75585fb7521dfa Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v1 2/2] pg_waldump: Report the position of a WAL read error When reading a record fails, pg_waldump prints "error in WAL record at %X/%08X" followed by the reader's message, using ReadRecPtr for the position. ReadRecPtr is the start of the last record read successfully, not the place where reading failed. At the end of WAL, the most common case, the prefix names the last valid record while the message names the end of WAL: error in WAL record at 0/017AC370: invalid record length at 0/017AC3F8: expected at least 24, got 0 Since 3f1ce97346 the same happens when a record is rejected while it is being decoded. Use EndRecPtr instead, the position where the failing record starts, as pg_rewind already does. Author: Rahul Yadav <rahul@rhyadav.dev> --- src/bin/pg_waldump/pg_waldump.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 6f0574a876..34ca41a7e6 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1521,7 +1521,7 @@ main(int argc, char **argv) if (errormsg) pg_fatal("error in WAL record at %X/%08X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), errormsg); /* -- 2.50.1 (Apple Git-155)
v1-0001-Report-the-decoded-record-s-LSN-in-DecodeXLogReco.patch (application/octet-stream, 4.4K) download From 542ff1da0f42ad21189418f33db68bf618a873e0 Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v1 1/2] Report the decoded record's LSN in DecodeXLogRecord() errors DecodeXLogRecord() is passed the LSN of the record it decodes, but its error messages printed state->ReadRecPtr instead. Before 3f1ce97346 added the circular WAL decoding buffer, the two were the same, since XLogReadRecord() set ReadRecPtr before decoding. Now records are decoded ahead of the one the reader last returned, so these messages named an earlier record: in pg_waldump the record just before the failing one, and in crash recovery, which reads further ahead, one several records back. Use the lsn argument instead. The only other caller, the WAL_DEBUG code in XLogInsertRecord(), passes the end of the record it has just inserted, which matches the LSN that code logs for it. Noticed while testing a fix for bug #19599. Author: Rahul Yadav <rahul@rhyadav.dev> Backpatch-through: 15 --- src/backend/access/transam/xlogreader.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7db7c273b0..6573270dab 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -1818,7 +1818,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "out-of-order block_id %u at %X/%08X", block_id, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } decoded->max_block_id = block_id; @@ -1841,7 +1841,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_HAS_DATA set, but no data included at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } if (!blk->has_data && blk->data_len != 0) @@ -1849,7 +1849,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPBLOCK_HAS_DATA not set, but data length is %d at %X/%08X", blk->data_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } datatotal += blk->data_len; @@ -1887,7 +1887,7 @@ DecodeXLogRecord(XLogReaderState *state, blk->hole_offset, blk->hole_length, blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1902,7 +1902,7 @@ DecodeXLogRecord(XLogReaderState *state, "BKPIMAGE_HAS_HOLE not set, but hole offset %d length %d at %X/%08X", blk->hole_offset, blk->hole_length, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1915,7 +1915,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPIMAGE_COMPRESSED set, but block image length %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1930,7 +1930,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_COMPRESSED set, but block image length is %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -1945,7 +1945,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_SAME_REL set but no previous rel at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1957,7 +1957,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "invalid block_id %u at %X/%08X", - block_id, LSN_FORMAT_ARGS(state->ReadRecPtr)); + block_id, LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -2024,7 +2024,7 @@ DecodeXLogRecord(XLogReaderState *state, shortdata_err: report_invalid_record(state, "record with invalid length at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); err: *errormsg = state->errormsg_buf; -- 2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
@ 2026-10-07 22:00 ` surya poondla <suryapoondla4@gmail.com>
  1 sibling, 0 replies; 9+ messages in thread

From: surya poondla @ 2026-10-07 22:00 UTC (permalink / raw)
  To: rahul@rhyadav.dev; +Cc: Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

Hi Rahul,

Thanks for the patches. I reviewed both, and tested 0001 on REL_19_STABLE
(3f5bfbce4c4) and 0001+0002 on master.

0001 looks right to me. The other checks in XLogDecodeNextRecord() already
report RecPtr; DecodeXLogRecord() is the only place that uses
ReadRecPtr, and that pointer has been unreliable there since 3f1ce97346.
You also correctly left RestoreBlockImage() alone, it only runs on a record
that has already been returned (redo, pg_waldump
--save-fullpage, pg_walinspect), so ReadRecPtr really is that record.

Thanks to the reproducing steps, I reproduced it the same way you did.
hole_offset set to 0 in an uncompressed FPI, CRC recomputed. I crashed a
cluster with kill -9
partway through a transaction after a checkpoint, then corrupted an FPI
from that transaction, at 0/01821288.

Crash recovery reported these LSNs (in all cases "redo done at 0/01821248"):

                      recovery_prefetch=try    recovery_prefetch=off
    unpatched         0/0180F608               0/01821248
    patched           0/01821288               0/01821288

With prefetching on, the unpatched message named the redo start point, 45
records (about 71kB) before the bad record.
The prefetcher had decoded that whole span in one go before redo got its
second record,
so ReadRecPtr was still at the redo point. So "several records back" in the
commit message undersells it a bit.
With recovery_prefetch on, the LSN can be anywhere in the read-ahead
window, and here it was the redo point. It might be worth saying that.

The patched and unpatched clusters ended up the same, i.e same stop point,
same rows, and byte-identical relation files (pg_control differed only
in timestamps). So 0001 changes only the message, which makes it an easy
backpatch.

One more case for pg_waldump, from a separate WAL segment (an initdb
template, not the cluster above) with the same corruption applied to
the FPI at 0/010165F8. When pg_waldump is started at the bad record (-s
0/010165F8), the unpatched build reports

    "could not find a valid record after 0/010165F8: ... at 0/01016568"

which is an LSN before the requested start. With 0001 it names 0/010165F8.

A few comments:
1. The WAL_DEBUG caller in XLogInsertRecord() passes EndPos, so after the
patch those messages show the end of the record rather than its
start. As you say, that matches what that code logs. It's still an
improvement, because debug_reader is never positioned, so the
messages printed 0/00000000 before. A line in the commit message would make
the difference from the other caller clear.

2. Could we have a test? pg_waldump's 001_basic.pl checks the "error in WAL
record at" message only for falling off the end of WAL, and
only matches the prefix. Patching two bytes of an FPI and fixing the CRC is
easy enough in Perl, and a check that the reported LSN equals
the record's LSN would cover both 0001 and 0002. I'm happy to share the
script I used if that helps.

3. pg_walinspect passes the same message through, so users of
pg_get_wal_records_info() will benefit from 0001 too.

Reaging the 0002:
+1. It does what pg_rewind's extractPageMap() and pg_walinspect's
ReadNextXLogRecord() already do when XLogReadRecord() fails, and with
the prefix and the message agree at the normal end of WAL. I agree about
keeping it to master.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
@ 2026-10-11 07:46 ` Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  1 sibling, 1 reply; 9+ messages in thread

From: Rahul @ 2026-10-11 07:46 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

Hi Surya,

Thanks for the review, and for testing on 19 and master.  Your
recovery_prefetch numbers show the problem much better than mine did.

v2 attached:

- 0001's commit message now says that with recovery_prefetch the
  message can name any record in the read-ahead window, with your
  redo start point as the example, and that pg_walinspect reports the
  same messages (your point 3).

- It also says the WAL_DEBUG messages now show the end of the
  inserted record, where they printed 0/00000000 before (point 1).

- 0001 adds a test to pg_waldump's 001_basic.pl (point 2).  It
  updates a row right after a checkpoint, so that the update record
  carries a full-page image with a hole, clears the hole offset in a
  copy of the WAL segment, and recomputes the record's CRC with
  crc32c().  It then checks that pg_waldump names that record, both
  when started before it and when started at it, your second
  pg_waldump case.  0002 extends it to check the "error in WAL record
  at" prefix too.  Without the fixes, those checks fail.

Thanks for offering your script.  Recomputing the CRC with crc32c()
kept the test short, but that function only exists from 18 on, so the
test can't go back further than that with 0001.

Regards,
Rahul Yadav

Attachments: download all (2 files, .tar.gz)

v2-0002-pg_waldump-Report-the-position-of-a-WAL-read-erro.patch (application/octet-stream, 2.4K) download From 9290d75aab3471e361529a6bef05d2a965f89775 Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v2 2/2] pg_waldump: Report the position of a WAL read error When reading a record fails, pg_waldump prints "error in WAL record at %X/%08X" followed by the reader's message, using ReadRecPtr for the position. ReadRecPtr is the start of the last record read successfully, not the place where reading failed. At the end of WAL, the most common case, the prefix names the last valid record while the message names the end of WAL: error in WAL record at 0/017AC370: invalid record length at 0/017AC3F8: expected at least 24, got 0 Since 3f1ce97346 the same happens when a record is rejected while it is being decoded. Use EndRecPtr instead, the position where the failing record starts, as pg_rewind's extractPageMap() and pg_walinspect already do when XLogReadRecord() fails. Extend the test added by the previous commit to check the prefix too. Author: Rahul Yadav <rahul@rhyadav.dev> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/P2qIkq4--F-9@rhyadav.dev --- src/bin/pg_waldump/pg_waldump.c | 2 +- src/bin/pg_waldump/t/001_basic.pl | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 6f0574a876..34ca41a7e6 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1521,7 +1521,7 @@ main(int argc, char **argv) if (errormsg) pg_fatal("error in WAL record at %X/%08X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), errormsg); /* diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index c345b07ef4..06044d58de 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -612,7 +612,7 @@ my $bad_fpi_error = qr{BKPIMAGE_HAS_HOLE set, but hole offset 0 length \d+ block image length \d+ at \Q$bad_lsn\E}; command_fails_like( [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_prev ], - qr{error in WAL record at [0-9A-F]+/[0-9A-F]+: $bad_fpi_error}, + qr{error in WAL record at \Q$bad_lsn\E: $bad_fpi_error}, 'decoding error names the rejected record'); command_fails_like( [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_lsn ], -- 2.50.1 (Apple Git-155)
v2-0001-Report-the-decoded-record-s-LSN-in-DecodeXLogReco.patch (application/octet-stream, 9.6K) download From d1017da5dd484418b9b784fa07e794ab28d32bdc Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v2 1/2] Report the decoded record's LSN in DecodeXLogRecord() errors DecodeXLogRecord() is passed the LSN of the record it decodes, but its error messages printed state->ReadRecPtr instead. Before 3f1ce97346 added the circular WAL decoding buffer, the two were the same, since XLogReadRecord() set ReadRecPtr before decoding. Now records are decoded ahead of the one the reader last returned, so these messages named an earlier record: in pg_waldump, and in recovery without prefetching, the record just before the failing one. With recovery_prefetch, it can be any record in the read-ahead window; in one test it was the redo start point, 45 records before the failing one. pg_walinspect reports the same messages. Use the lsn argument instead. The only other caller, the WAL_DEBUG code in XLogInsertRecord(), passes the end of the record it has just inserted, which matches the LSN that code logs for it. Its messages printed 0/00000000 before, as that reader is never positioned. Add a test that corrupts a full-page image so that its record fails decoding, and checks that pg_waldump's error names that record. Noticed while testing a fix for bug #19599. Author: Rahul Yadav <rahul@rhyadav.dev> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/P2qIkq4--F-9@rhyadav.dev Backpatch-through: 15 --- src/backend/access/transam/xlogreader.c | 20 ++--- src/bin/pg_waldump/t/001_basic.pl | 110 ++++++++++++++++++++++++ 2 files changed, 120 insertions(+), 10 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 1db5ca48c3..9f53c132a6 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -1888,7 +1888,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "out-of-order block_id %u at %X/%08X", block_id, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } decoded->max_block_id = block_id; @@ -1911,7 +1911,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_HAS_DATA set, but no data included at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } if (!blk->has_data && blk->data_len != 0) @@ -1919,7 +1919,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPBLOCK_HAS_DATA not set, but data length is %d at %X/%08X", blk->data_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } datatotal += blk->data_len; @@ -1957,7 +1957,7 @@ DecodeXLogRecord(XLogReaderState *state, blk->hole_offset, blk->hole_length, blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1972,7 +1972,7 @@ DecodeXLogRecord(XLogReaderState *state, "BKPIMAGE_HAS_HOLE not set, but hole offset %d length %d at %X/%08X", blk->hole_offset, blk->hole_length, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1985,7 +1985,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPIMAGE_COMPRESSED set, but block image length %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -2000,7 +2000,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_COMPRESSED set, but block image length is %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -2015,7 +2015,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_SAME_REL set but no previous rel at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -2027,7 +2027,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "invalid block_id %u at %X/%08X", - block_id, LSN_FORMAT_ARGS(state->ReadRecPtr)); + block_id, LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -2094,7 +2094,7 @@ DecodeXLogRecord(XLogReaderState *state, shortdata_err: report_invalid_record(state, "record with invalid length at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); err: *errormsg = state->errormsg_buf; diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index 7b33efc629..c345b07ef4 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -509,4 +509,114 @@ for my $scenario (@scenarios) } } +# Check the LSN reported for a record that is rejected while it is decoded. +# Update a row right after a checkpoint, so that the update record carries a +# full-page image of a page with a hole. In a copy of the WAL segment, clear +# the image's hole offset and recompute the record's CRC, so that the record +# passes the CRC check but fails DecodeXLogRecord()'s checks. +$node->start; +$node->safe_psql('postgres', + 'CREATE TABLE t_bad_fpi (a int); INSERT INTO t_bad_fpi VALUES (1);'); +my $bad_fpi_rel = join('/', + $default_ts_oid, $postgres_db_oid, + $node->safe_psql('postgres', q{SELECT pg_relation_filenode('t_bad_fpi')})); +my $wal_block_size = $node->safe_psql('postgres', 'SHOW wal_block_size'); +my $wal_segment_size = $node->safe_psql('postgres', + q{SELECT setting FROM pg_settings WHERE name = 'wal_segment_size'}); + +# Retry until the record fits on one WAL page and doesn't start right after a +# page header, so that it can be patched in place and starts where the +# previous record ends. +my ($bad_lsn, $bad_prev, $bad_len, $bad_off); +foreach my $try (1 .. 10) +{ + my $from = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $node->safe_psql('postgres', 'CHECKPOINT; UPDATE t_bad_fpi SET a = a + 1;'); + my $to = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + my ($stdout, $stderr); + IPC::Run::run [ + 'pg_waldump', + '--path' => $node->data_dir . '/pg_wal', + '--start' => $from, + '--end' => $to, + '--relation' => $bad_fpi_rel, + '--fullpage' + ], + '>' => \$stdout, + '2>' => \$stderr; + next + unless $stdout =~ + m{len \(rec/tot\):\s*\d+/\s*(\d+), .* lsn: ([0-9A-F]+/[0-9A-F]+), prev ([0-9A-F]+/[0-9A-F]+)}; + ($bad_len, $bad_lsn, $bad_prev) = ($1, $2, $3); + + # The segment and page sizes divide 2^32, so the low half is enough. + my $lsn_lo = hex((split m{/}, $bad_lsn)[1]); + my $page_off = $lsn_lo % $wal_block_size; + $bad_off = $lsn_lo % $wal_segment_size; + last if $page_off > 40 && $page_off + $bad_len <= $wal_block_size; + undef $bad_lsn; +} +ok(defined $bad_lsn, 'found a record with a full-page image to corrupt'); + +my $bad_walfile = + $node->safe_psql('postgres', "SELECT pg_walfile_name('$bad_lsn')"); +my $rec; +open(my $wal_fh, '<', $node->data_dir . "/pg_wal/$bad_walfile") + or BAIL_OUT("open failed: $!"); +binmode $wal_fh; +sysseek($wal_fh, $bad_off, 0) + or BAIL_OUT("sysseek failed: $!"); +sysread($wal_fh, $rec, $bad_len) == $bad_len + or BAIL_OUT("sysread failed: $!"); +close($wal_fh); + +# The record's CRC covers the data after the 24-byte header, followed by the +# header up to xl_crc. +sub record_crc +{ + my ($rec) = @_; + my $hex = unpack('H*', substr($rec, 24) . substr($rec, 0, 20)); + return $node->safe_psql('postgres', "SELECT crc32c('\\x$hex'::bytea)"); +} +is(record_crc($rec), unpack('V', substr($rec, 20, 4)), + 'CRC computed for the original record'); + +# The first block header (id, fork_flags, data_length) is followed by the +# image header (length, hole_offset, bimg_info). +my ($block_id, $fork_flags, undef, undef, undef, $bimg_info) = + unpack('C C v v v C', substr($rec, 24, 9)); +ok( $block_id == 0 + && ($fork_flags & 0x10) # BKPBLOCK_HAS_IMAGE + && ($bimg_info & 0x01) # BKPIMAGE_HAS_HOLE + && !($bimg_info & 0x1C), # not compressed + 'block 0 carries an uncompressed full-page image with a hole'); + +substr($rec, 30, 2) = pack('v', 0); # hole_offset +substr($rec, 20, 4) = pack('V', record_crc($rec)); +$node->stop; + +my $bad_wal_dir = PostgreSQL::Test::Utils::tempdir_short(); +copy($node->data_dir . "/pg_wal/$bad_walfile", "$bad_wal_dir/$bad_walfile") + || die "copying $bad_walfile $!"; +open($wal_fh, '+<', "$bad_wal_dir/$bad_walfile") + or BAIL_OUT("open failed: $!"); +binmode $wal_fh; +sysseek($wal_fh, $bad_off, 0) + or BAIL_OUT("sysseek failed: $!"); +syswrite($wal_fh, $rec) == $bad_len + or BAIL_OUT("syswrite failed: $!"); +close($wal_fh) + or BAIL_OUT("close failed: $!"); + +my $bad_fpi_error = + qr{BKPIMAGE_HAS_HOLE set, but hole offset 0 length \d+ block image length \d+ at \Q$bad_lsn\E}; +command_fails_like( + [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_prev ], + qr{error in WAL record at [0-9A-F]+/[0-9A-F]+: $bad_fpi_error}, + 'decoding error names the rejected record'); +command_fails_like( + [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_lsn ], + qr{could not find a valid record after \Q$bad_lsn\E: $bad_fpi_error}, + 'decoding error names the rejected record when starting at it'); + done_testing(); -- 2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
@ 2026-10-12 18:07   ` surya poondla <suryapoondla4@gmail.com>
  2026-10-30 14:59     ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 9+ messages in thread

From: surya poondla @ 2026-10-12 18:07 UTC (permalink / raw)
  To: Rahul <rahul@rhyadav.dev>; +Cc: Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

Hi Rahul,

Thanks for v2. It addresses everything I raised.

I tested it on REL_19_STABLE (1a1248818fb) and master (9e7d152488f).
Both patches apply cleanly and build without warnings, and the code
changes are the same as in v1.  The pg_waldump TAP tests pass, and the new
checks behave as you described:

All tests pass with v2.

without 0002's C change:    only the "error in WAL record at" prefix check
fails
without both C changes:     both LSN checks fail

Without the fixes, the start-at-the-bad-record case again names an LSN
before the requested start point (0/017B2B78 when starting at
0/017B2C00 on master), so the test covers that case too.

The commit messages look good to me.  As you said, the test needs crc32c(),
which only exists from 18 on, so it can't go back further
than 18, while the code fix goes back to 15.

The patch looks ready for a Committer.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
@ 2026-10-30 14:59     ` Fujii Masao <masao.fujii@gmail.com>
  2026-10-30 15:52       ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 9+ messages in thread

From: Fujii Masao @ 2026-10-30 14:59 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: Rahul <rahul@rhyadav.dev>; Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

On Tue, Oct 13, 2026 at 3:07 AM surya poondla <suryapoondla4@gmail.com> wrote:
> The patch looks ready for a Committer.

DecodeXLogRecord() changes in 0001 patch look good to me.

But, do we really need a new test for such a small fix? Generating and
locating a record, handling boundaries, modifying WAL, recomputing the CRC,
and copying the segment seem like a lot of complexity and maintenance
overhead just to verify the reported LSN, I'm afraid.

If we really want this, seems it's better to consider it separately as a test
for malformed WAL detection, including the LSN check. I'm just not sure
this additional infrastructure is worth the effort, though..

Regards,


-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-10-30 14:59     ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
@ 2026-10-30 15:52       ` surya poondla <suryapoondla4@gmail.com>
  2026-11-02 01:44         ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 9+ messages in thread

From: surya poondla @ 2026-10-30 15:52 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Rahul <rahul@rhyadav.dev>; Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

Hi Fujii,


> But, do we really need a new test for such a small fix?
>

That's fair. I was the one who asked for the test, but I agree it's a
lot of test code to check an LSN in an error message. I'm happy for 0001 to
go in without it.


Also, what do you think about 0002? It makes pg_waldump's "error in WAL
record at" prefix show where reading failed, instead of the last
good record. pg_rewind's extractPageMap() and pg_walinspect already
work this way. We could make this master only, since it changes output
people may be used to.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-10-30 14:59     ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  2026-10-30 15:52       ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
@ 2026-11-02 01:44         ` Fujii Masao <masao.fujii@gmail.com>
  2026-11-02 04:49           ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 9+ messages in thread

From: Fujii Masao @ 2026-11-02 01:44 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: Rahul <rahul@rhyadav.dev>; Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

On Sat, Oct 31, 2026 at 12:52 AM surya poondla <suryapoondla4@gmail.com> wrote:
> Also, what do you think about 0002? It makes pg_waldump's "error in WAL record at" prefix show where reading failed, instead of the last
> good record.

EndRecPtr is the starting position for the next read, which does not
always correspond to the start of the failing record. No? For example,
if decoding fails on the first record after a page header, EndRecPtr
still points to the start of the page? So if the goal is to report
the exact location of the failing record, isn't this change insufficient?

OTOH, if the goal is consistency with pg_rewind and pg_walinspect,
I'm fine with using EndRecPtr. But, as with 0001, I don't think the test
change in 0002 is necessary.

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-10-30 14:59     ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  2026-10-30 15:52       ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-11-02 01:44         ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
@ 2026-11-02 04:49           ` surya poondla <suryapoondla4@gmail.com>
  2026-11-03 01:57             ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 9+ messages in thread

From: surya poondla @ 2026-11-02 04:49 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Rahul <rahul@rhyadav.dev>; Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

Hi Fujii,

EndRecPtr is the starting position for the next read, which does not
> always correspond to the start of the failing record. No? For example,
> if decoding fails on the first record after a page header, EndRecPtr
> still points to the start of the page?
>

Yes, you're right. When the next record starts right after a page header,
 EndRecPtr points to the start of the page, so the prefix would be off
  by the header size there. The LSN at the end of the message, from
0001, is still exact, since XLogDecodeNextRecord() skips any page header
before passing RecPtr to
DecodeXLogRecord().


> OTOH, if the goal is consistency with pg_rewind and pg_walinspect,
> I'm fine with using EndRecPtr. But, as with 0001, I don't think the test
> change in 0002 is necessary.


0002  is mainly a consistency change with pg_rewind and pg_walinspect.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Wrong LSN in WAL decoding error messages
  2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
  2026-10-11 07:46 ` Re: Wrong LSN in WAL decoding error messages Rahul <rahul@rhyadav.dev>
  2026-10-12 18:07   ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-10-30 14:59     ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  2026-10-30 15:52       ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
  2026-11-02 01:44         ` Re: Wrong LSN in WAL decoding error messages Fujii Masao <masao.fujii@gmail.com>
  2026-11-02 04:49           ` Re: Wrong LSN in WAL decoding error messages surya poondla <suryapoondla4@gmail.com>
@ 2026-11-03 01:57             ` Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Fujii Masao @ 2026-11-03 01:57 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: Rahul <rahul@rhyadav.dev>; Pgsql Hackers <pgsql-hackers@lists.postgresql.org>; Thomas Munro <thomas.munro@gmail.com>; Ju Grigorev <ju.grigorev@ftdata.ru>

On Mon, Nov 2, 2026 at 1:49 PM surya poondla <suryapoondla4@gmail.com> wrote:
> 0002  is mainly a consistency change with pg_rewind and pg_walinspect.

I've pushed both patches. Thanks!

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 9+ messages in thread


end of thread, other threads:[~2026-11-03 01:57 UTC | newest]

Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-01 07:17 Wrong LSN in WAL decoding error messages rahul@rhyadav.dev
2026-10-07 22:00 ` surya poondla <suryapoondla4@gmail.com>
2026-10-11 07:46 ` Rahul <rahul@rhyadav.dev>
2026-10-12 18:07   ` surya poondla <suryapoondla4@gmail.com>
2026-10-30 14:59     ` Fujii Masao <masao.fujii@gmail.com>
2026-10-30 15:52       ` surya poondla <suryapoondla4@gmail.com>
2026-11-02 01:44         ` Fujii Masao <masao.fujii@gmail.com>
2026-11-02 04:49           ` surya poondla <suryapoondla4@gmail.com>
2026-11-03 01:57             ` Fujii Masao <masao.fujii@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox