pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Rahul <rahul@rhyadav.dev>
To: surya poondla <suryapoondla4@gmail.com>
Cc: Pgsql Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Ju Grigorev <ju.grigorev@ftdata.ru>
Subject: Re: Wrong LSN in WAL decoding error messages
Date: Sun, 11 Oct 2026 09:46:44 +0200 (CEST)
Message-ID: <P3dvLyJ--F-9@rhyadav.dev> (raw)
In-Reply-To: <CAOVWO5rGVkiB-MUxKJVtUv3L=H3taWx7Rsg5_zea-nmQknU-MQ@mail.gmail.com-P3NNjNA----9>
References: <P2qIkq4--F-9@rhyadav.dev>
	<CAOVWO5rGVkiB-MUxKJVtUv3L=H3taWx7Rsg5_zea-nmQknU-MQ@mail.gmail.com-P3NNjNA----9>

Hi Surya,

Thanks for the review, and for testing on 19 and master.  Your
recovery_prefetch numbers show the problem much better than mine did.

v2 attached:

- 0001's commit message now says that with recovery_prefetch the
  message can name any record in the read-ahead window, with your
  redo start point as the example, and that pg_walinspect reports the
  same messages (your point 3).

- It also says the WAL_DEBUG messages now show the end of the
  inserted record, where they printed 0/00000000 before (point 1).

- 0001 adds a test to pg_waldump's 001_basic.pl (point 2).  It
  updates a row right after a checkpoint, so that the update record
  carries a full-page image with a hole, clears the hole offset in a
  copy of the WAL segment, and recomputes the record's CRC with
  crc32c().  It then checks that pg_waldump names that record, both
  when started before it and when started at it, your second
  pg_waldump case.  0002 extends it to check the "error in WAL record
  at" prefix too.  Without the fixes, those checks fail.

Thanks for offering your script.  Recomputing the CRC with crc32c()
kept the test short, but that function only exists from 18 on, so the
test can't go back further than that with 0001.

Regards,
Rahul Yadav

Attachments: download all (2 files, .tar.gz)

v2-0002-pg_waldump-Report-the-position-of-a-WAL-read-erro.patch (application/octet-stream, 2.4K) download From 9290d75aab3471e361529a6bef05d2a965f89775 Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v2 2/2] pg_waldump: Report the position of a WAL read error When reading a record fails, pg_waldump prints "error in WAL record at %X/%08X" followed by the reader's message, using ReadRecPtr for the position. ReadRecPtr is the start of the last record read successfully, not the place where reading failed. At the end of WAL, the most common case, the prefix names the last valid record while the message names the end of WAL: error in WAL record at 0/017AC370: invalid record length at 0/017AC3F8: expected at least 24, got 0 Since 3f1ce97346 the same happens when a record is rejected while it is being decoded. Use EndRecPtr instead, the position where the failing record starts, as pg_rewind's extractPageMap() and pg_walinspect already do when XLogReadRecord() fails. Extend the test added by the previous commit to check the prefix too. Author: Rahul Yadav <rahul@rhyadav.dev> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/P2qIkq4--F-9@rhyadav.dev --- src/bin/pg_waldump/pg_waldump.c | 2 +- src/bin/pg_waldump/t/001_basic.pl | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 6f0574a876..34ca41a7e6 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1521,7 +1521,7 @@ main(int argc, char **argv) if (errormsg) pg_fatal("error in WAL record at %X/%08X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), errormsg); /* diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index c345b07ef4..06044d58de 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -612,7 +612,7 @@ my $bad_fpi_error = qr{BKPIMAGE_HAS_HOLE set, but hole offset 0 length \d+ block image length \d+ at \Q$bad_lsn\E}; command_fails_like( [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_prev ], - qr{error in WAL record at [0-9A-F]+/[0-9A-F]+: $bad_fpi_error}, + qr{error in WAL record at \Q$bad_lsn\E: $bad_fpi_error}, 'decoding error names the rejected record'); command_fails_like( [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_lsn ], -- 2.50.1 (Apple Git-155)
v2-0001-Report-the-decoded-record-s-LSN-in-DecodeXLogReco.patch (application/octet-stream, 9.6K) download From d1017da5dd484418b9b784fa07e794ab28d32bdc Mon Sep 17 00:00:00 2001 From: Rahul Yadav <rahul@rhyadav.dev> Date: Thu, 1 Oct 2026 07:46:25 +0100 Subject: [PATCH v2 1/2] Report the decoded record's LSN in DecodeXLogRecord() errors DecodeXLogRecord() is passed the LSN of the record it decodes, but its error messages printed state->ReadRecPtr instead. Before 3f1ce97346 added the circular WAL decoding buffer, the two were the same, since XLogReadRecord() set ReadRecPtr before decoding. Now records are decoded ahead of the one the reader last returned, so these messages named an earlier record: in pg_waldump, and in recovery without prefetching, the record just before the failing one. With recovery_prefetch, it can be any record in the read-ahead window; in one test it was the redo start point, 45 records before the failing one. pg_walinspect reports the same messages. Use the lsn argument instead. The only other caller, the WAL_DEBUG code in XLogInsertRecord(), passes the end of the record it has just inserted, which matches the LSN that code logs for it. Its messages printed 0/00000000 before, as that reader is never positioned. Add a test that corrupts a full-page image so that its record fails decoding, and checks that pg_waldump's error names that record. Noticed while testing a fix for bug #19599. Author: Rahul Yadav <rahul@rhyadav.dev> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/P2qIkq4--F-9@rhyadav.dev Backpatch-through: 15 --- src/backend/access/transam/xlogreader.c | 20 ++--- src/bin/pg_waldump/t/001_basic.pl | 110 ++++++++++++++++++++++++ 2 files changed, 120 insertions(+), 10 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 1db5ca48c3..9f53c132a6 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -1888,7 +1888,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "out-of-order block_id %u at %X/%08X", block_id, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } decoded->max_block_id = block_id; @@ -1911,7 +1911,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_HAS_DATA set, but no data included at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } if (!blk->has_data && blk->data_len != 0) @@ -1919,7 +1919,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPBLOCK_HAS_DATA not set, but data length is %d at %X/%08X", blk->data_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } datatotal += blk->data_len; @@ -1957,7 +1957,7 @@ DecodeXLogRecord(XLogReaderState *state, blk->hole_offset, blk->hole_length, blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1972,7 +1972,7 @@ DecodeXLogRecord(XLogReaderState *state, "BKPIMAGE_HAS_HOLE not set, but hole offset %d length %d at %X/%08X", blk->hole_offset, blk->hole_length, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -1985,7 +1985,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "BKPIMAGE_COMPRESSED set, but block image length %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -2000,7 +2000,7 @@ DecodeXLogRecord(XLogReaderState *state, report_invalid_record(state, "neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_COMPRESSED set, but block image length is %d at %X/%08X", blk->bimg_len, - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -2015,7 +2015,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "BKPBLOCK_SAME_REL set but no previous rel at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); goto err; } @@ -2027,7 +2027,7 @@ DecodeXLogRecord(XLogReaderState *state, { report_invalid_record(state, "invalid block_id %u at %X/%08X", - block_id, LSN_FORMAT_ARGS(state->ReadRecPtr)); + block_id, LSN_FORMAT_ARGS(lsn)); goto err; } } @@ -2094,7 +2094,7 @@ DecodeXLogRecord(XLogReaderState *state, shortdata_err: report_invalid_record(state, "record with invalid length at %X/%08X", - LSN_FORMAT_ARGS(state->ReadRecPtr)); + LSN_FORMAT_ARGS(lsn)); err: *errormsg = state->errormsg_buf; diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index 7b33efc629..c345b07ef4 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -509,4 +509,114 @@ for my $scenario (@scenarios) } } +# Check the LSN reported for a record that is rejected while it is decoded. +# Update a row right after a checkpoint, so that the update record carries a +# full-page image of a page with a hole. In a copy of the WAL segment, clear +# the image's hole offset and recompute the record's CRC, so that the record +# passes the CRC check but fails DecodeXLogRecord()'s checks. +$node->start; +$node->safe_psql('postgres', + 'CREATE TABLE t_bad_fpi (a int); INSERT INTO t_bad_fpi VALUES (1);'); +my $bad_fpi_rel = join('/', + $default_ts_oid, $postgres_db_oid, + $node->safe_psql('postgres', q{SELECT pg_relation_filenode('t_bad_fpi')})); +my $wal_block_size = $node->safe_psql('postgres', 'SHOW wal_block_size'); +my $wal_segment_size = $node->safe_psql('postgres', + q{SELECT setting FROM pg_settings WHERE name = 'wal_segment_size'}); + +# Retry until the record fits on one WAL page and doesn't start right after a +# page header, so that it can be patched in place and starts where the +# previous record ends. +my ($bad_lsn, $bad_prev, $bad_len, $bad_off); +foreach my $try (1 .. 10) +{ + my $from = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $node->safe_psql('postgres', 'CHECKPOINT; UPDATE t_bad_fpi SET a = a + 1;'); + my $to = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + my ($stdout, $stderr); + IPC::Run::run [ + 'pg_waldump', + '--path' => $node->data_dir . '/pg_wal', + '--start' => $from, + '--end' => $to, + '--relation' => $bad_fpi_rel, + '--fullpage' + ], + '>' => \$stdout, + '2>' => \$stderr; + next + unless $stdout =~ + m{len \(rec/tot\):\s*\d+/\s*(\d+), .* lsn: ([0-9A-F]+/[0-9A-F]+), prev ([0-9A-F]+/[0-9A-F]+)}; + ($bad_len, $bad_lsn, $bad_prev) = ($1, $2, $3); + + # The segment and page sizes divide 2^32, so the low half is enough. + my $lsn_lo = hex((split m{/}, $bad_lsn)[1]); + my $page_off = $lsn_lo % $wal_block_size; + $bad_off = $lsn_lo % $wal_segment_size; + last if $page_off > 40 && $page_off + $bad_len <= $wal_block_size; + undef $bad_lsn; +} +ok(defined $bad_lsn, 'found a record with a full-page image to corrupt'); + +my $bad_walfile = + $node->safe_psql('postgres', "SELECT pg_walfile_name('$bad_lsn')"); +my $rec; +open(my $wal_fh, '<', $node->data_dir . "/pg_wal/$bad_walfile") + or BAIL_OUT("open failed: $!"); +binmode $wal_fh; +sysseek($wal_fh, $bad_off, 0) + or BAIL_OUT("sysseek failed: $!"); +sysread($wal_fh, $rec, $bad_len) == $bad_len + or BAIL_OUT("sysread failed: $!"); +close($wal_fh); + +# The record's CRC covers the data after the 24-byte header, followed by the +# header up to xl_crc. +sub record_crc +{ + my ($rec) = @_; + my $hex = unpack('H*', substr($rec, 24) . substr($rec, 0, 20)); + return $node->safe_psql('postgres', "SELECT crc32c('\\x$hex'::bytea)"); +} +is(record_crc($rec), unpack('V', substr($rec, 20, 4)), + 'CRC computed for the original record'); + +# The first block header (id, fork_flags, data_length) is followed by the +# image header (length, hole_offset, bimg_info). +my ($block_id, $fork_flags, undef, undef, undef, $bimg_info) = + unpack('C C v v v C', substr($rec, 24, 9)); +ok( $block_id == 0 + && ($fork_flags & 0x10) # BKPBLOCK_HAS_IMAGE + && ($bimg_info & 0x01) # BKPIMAGE_HAS_HOLE + && !($bimg_info & 0x1C), # not compressed + 'block 0 carries an uncompressed full-page image with a hole'); + +substr($rec, 30, 2) = pack('v', 0); # hole_offset +substr($rec, 20, 4) = pack('V', record_crc($rec)); +$node->stop; + +my $bad_wal_dir = PostgreSQL::Test::Utils::tempdir_short(); +copy($node->data_dir . "/pg_wal/$bad_walfile", "$bad_wal_dir/$bad_walfile") + || die "copying $bad_walfile $!"; +open($wal_fh, '+<', "$bad_wal_dir/$bad_walfile") + or BAIL_OUT("open failed: $!"); +binmode $wal_fh; +sysseek($wal_fh, $bad_off, 0) + or BAIL_OUT("sysseek failed: $!"); +syswrite($wal_fh, $rec) == $bad_len + or BAIL_OUT("syswrite failed: $!"); +close($wal_fh) + or BAIL_OUT("close failed: $!"); + +my $bad_fpi_error = + qr{BKPIMAGE_HAS_HOLE set, but hole offset 0 length \d+ block image length \d+ at \Q$bad_lsn\E}; +command_fails_like( + [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_prev ], + qr{error in WAL record at [0-9A-F]+/[0-9A-F]+: $bad_fpi_error}, + 'decoding error names the rejected record'); +command_fails_like( + [ 'pg_waldump', '--path' => $bad_wal_dir, '--start' => $bad_lsn ], + qr{could not find a valid record after \Q$bad_lsn\E: $bad_fpi_error}, + 'decoding error names the rejected record when starting at it'); + done_testing(); -- 2.50.1 (Apple Git-155)

view thread (9+ messages)  latest in thread

Message-ID: <P3dvLyJ--F-9@rhyadav.dev>
Permalink:  ../P3dvLyJ--F-9@rhyadav.dev/
Also on:    postgresql.org/message-id/P3dvLyJ--F-9@rhyadav.dev

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: rahul@rhyadav.dev, suryapoondla4@gmail.com, thomas.munro@gmail.com, ju.grigorev@ftdata.ru
  Subject: Re: Wrong LSN in WAL decoding error messages
  In-Reply-To: <P3dvLyJ--F-9@rhyadav.dev>
  References: <P2qIkq4--F-9@rhyadav.dev> <CAOVWO5rGVkiB-MUxKJVtUv3L=H3taWx7Rsg5_zea-nmQknU-MQ@mail.gmail.com-P3NNjNA----9> <P3dvLyJ--F-9@rhyadav.dev>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox