agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v14] Make End-Of-Recovery error less scary 39+ messages / 2 participants [nested] [flat]
* [PATCH v15] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 125 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 92 ++++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++++ 6 files changed, 297 insertions(+), 47 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..ba1c1ece87 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -42,6 +42,8 @@ static bool allocate_recordbuf(XLogReaderState *state, uint32 reclength); static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -121,6 +123,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +295,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -380,12 +384,11 @@ restart: * whole header. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Otherwise do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. @@ -399,18 +402,13 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ; if (total_len > len) { @@ -588,6 +586,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -719,6 +726,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -730,14 +791,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (record->xl_rmid > RM_MAX_ID) { report_invalid_record(state, @@ -836,6 +892,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +1002,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index f9f212680b..750056acaf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1592,7 +1592,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -1706,7 +1706,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1765,13 +1765,20 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that indicate @@ -2939,6 +2946,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -2954,6 +2962,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2963,13 +2983,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -3000,11 +3023,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3017,11 +3043,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3042,12 +3073,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3129,12 +3172,16 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return -1; } } @@ -3767,7 +3814,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ceaff097b9..4f117ea4da 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 2340dc247b..215abe95dc 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1173,9 +1173,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..7b314ef10e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 14154d1ce0..01033334d6 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,7 +10,9 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -48,7 +50,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -61,4 +71,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Wed_Mar__2_11_17_04_2022_065)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v7] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 89 +++++++++++++++++++------ src/backend/access/transam/xlogreader.c | 42 ++++++++++++ src/backend/replication/walreceiver.c | 3 +- src/include/access/xlogreader.h | 1 + 4 files changed, 112 insertions(+), 23 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index d894af310a..fa435faec4 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4469,6 +4469,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4484,6 +4485,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = + xlogreader->ReadRecPtr ? + xlogreader->ReadRecPtr : xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4493,12 +4506,11 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error message + * right now. Otherwise the message if any is shown as a part of + * the end-of-WAL message below. */ - if (errormsg) + if (!xlogreader->EndOfWAL && errormsg) ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4530,11 +4542,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4547,11 +4560,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4599,12 +4618,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7536,7 +7576,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7774,7 +7814,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7834,13 +7874,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13130,7 +13177,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 3a7de02565..e16b6fe041 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,16 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the messages is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message at + * it should be more detailed. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +742,36 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the whole + * header is zeroed. + */ + char *p = (char *)record; + char *pe = (char *)record + SizeOfXLogRecord; + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* it is completely zeroed, call it a day */ + report_invalid_record(state, "empty record header found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + } + else + { + /* Otherwise we found a garbage header.. */ + report_invalid_record(state, "garbage record header at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + } + + return false; + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7a7eb3784e..ba3c4bd550 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,8 +471,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index de6fd791fe..1241b85838 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the -- 2.27.0 ----Next_Part(Wed_Dec__8_16_01_47_2021_480)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v14] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 78 ++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 92 ++++++++++++++----- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++++++++ 6 files changed, 268 insertions(+), 29 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..22982c4de7 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,40 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p; + char *pe; + + /* scan from the beginning of the record to the end of block */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +881,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +991,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index f9f212680b..750056acaf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1592,7 +1592,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -1706,7 +1706,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1765,13 +1765,20 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that indicate @@ -2939,6 +2946,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -2954,6 +2962,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2963,13 +2983,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -3000,11 +3023,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3017,11 +3043,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3042,12 +3073,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3129,12 +3172,16 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return -1; } } @@ -3767,7 +3814,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ceaff097b9..4f117ea4da 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..7b314ef10e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 14154d1ce0..01033334d6 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,7 +10,9 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -48,7 +50,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -61,4 +71,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Thu_Feb_17_16_50_01_2022_997)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v2] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Make this message less scary as "reached end of WAL". --- src/backend/access/transam/xlog.c | 72 ++++++++++++++++++++------- src/backend/replication/walreceiver.c | 3 +- 2 files changed, 55 insertions(+), 20 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index d19408b3be..849cf6fe6b 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4282,12 +4282,15 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); ReadRecPtr = xlogreader->ReadRecPtr; EndRecPtr = xlogreader->EndRecPtr; if (record == NULL) { + ErrRecPtr = ReadRecPtr ? ReadRecPtr : EndRecPtr; + if (readFile >= 0) { close(readFile); @@ -4295,14 +4298,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we are fetching checkpoint, we emit the error message right + * now. Otherwise the error is regarded as "end of WAL" and the + * message if any is shown as a part of the end-of-WAL message + * below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, EndRecPtr), + if (fetching_ckpt && errormsg) + { + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); + } } /* @@ -4332,11 +4337,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4349,11 +4355,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + (uint32) (ErrRecPtr >> 32), + (uint32) ErrRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4391,12 +4404,35 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * We reached the end of WAL, show the messages just once at the + * same LSN. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg (fmt, (uint32) (EndRecPtr >> 32), + (uint32) EndRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 2ab15c3cbb..682dbb4e1f 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -478,8 +478,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, (uint32) (LogstreamResult.Write >> 32), (uint32) LogstreamResult.Write))); endofwal = true; -- 2.18.2 ----Next_Part(Thu_Mar__5_16_06_50_2020_335)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Make this message less scary as "reached end of WAL". --- src/backend/access/transam/xlog.c | 45 ++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index d19408b3be..452c376f62 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4288,6 +4288,10 @@ ReadRecord(XLogReaderState *xlogreader, int emode, EndRecPtr = xlogreader->EndRecPtr; if (record == NULL) { + int actual_emode = + emode_for_corrupt_record(emode, + ReadRecPtr ? ReadRecPtr : EndRecPtr); + if (readFile >= 0) { close(readFile); @@ -4295,14 +4299,41 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * randAccess here means we are reading successive records during + * recovery. If we get here during recovery, we can assume that we + * reached the end of WAL. Otherwise something's really wrong and + * we report just only the errormsg if any. If we don't receive + * errormsg here, we already logged something. We don't emit + * "reached end of WAL" in muted messages. + * + * Note: errormsg is alreay translated. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!private->randAccess && actual_emode == emode) + { + if (StandbyMode) + ereport(actual_emode, + (errmsg ("rached end of WAL at %X/%X on timeline %u in %s during streaming replication", + (uint32) (EndRecPtr >> 32), (uint32) EndRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + else if (InArchiveRecovery) + ereport(actual_emode, + (errmsg ("rached end of WAL at %X/%X on timeline %u in %s during archive recovery", + (uint32) (EndRecPtr >> 32), (uint32) EndRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + else + ereport(actual_emode, + (errmsg ("rached end of WAL at %X/%X on timeline %u in %s during crash recovery", + (uint32) (EndRecPtr >> 32), (uint32) EndRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } + else if (errormsg) + ereport(actual_emode, (errmsg_internal("%s", errormsg))); } /* -- 2.18.2 ----Next_Part(Fri_Feb_28_16_01_00_2020_273)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v12] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 92 +++++++++++++----- src/backend/access/transam/xlogreader.c | 78 ++++++++++++++++ src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 108 +++++++++++++++++++++- 6 files changed, 268 insertions(+), 31 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 958220c495..bf1d40e7cb 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4557,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4577,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4635,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7581,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7819,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7879,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -12434,12 +12478,14 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; - + xlogreader->EndOfWAL = true; return -1; } } @@ -13097,7 +13143,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..22982c4de7 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,40 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p; + char *pe; + + /* scan from the beginning of the record to the end of block */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +881,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +991,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..1a7a692bc0 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,10 +471,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..7b314ef10e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..1d7476c309 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,99 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Thu_Feb_10_15_17_36_2022_468)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v9] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 91 +++++++++++++----- src/backend/access/transam/xlogreader.c | 61 ++++++++++++ src/backend/replication/walreceiver.c | 3 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 110 +++++++++++++++++++++- 6 files changed, 251 insertions(+), 28 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index dfe2a0bcce..5727e0939f 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at the + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = + xlogreader->ReadRecPtr ? + xlogreader->ReadRecPtr : xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error message + * right now. Otherwise the message if any is shown as a part of + * the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4553,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4571,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4629,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7584,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7822,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7882,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13097,7 +13144,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..418fb66ef2 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,31 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the whole + * header is filled with zeroes. + */ + char *p = (char *)record; + char *pe = (char *)record + SizeOfXLogRecord; + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* it is completely zeroed, call it a day */ + report_invalid_record(state, "empty record header found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +872,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0 ; i < XLOG_BLCKSZ && phdr[i] == 0 ; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..3034f8281e 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,8 +471,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..3eeba220a1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..67d264df26 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,101 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +my $chkptfile; +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# and the end-of-wal messages shouldn't be seen +# the same message has been confirmed in the past +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Thu_Jan_27_10_35_47_2022_592)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v10] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 91 +++++++++++++----- src/backend/access/transam/xlogreader.c | 77 +++++++++++++++ src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 110 +++++++++++++++++++++- 6 files changed, 269 insertions(+), 30 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index dfe2a0bcce..378c13ccf7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at the + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = + xlogreader->ReadRecPtr ? + xlogreader->ReadRecPtr : xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error + * message right now. Otherwise the message if any is shown as a + * part of the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4553,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4571,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4629,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7584,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7822,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7882,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13097,7 +13144,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..9bcc4a2d37 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,39 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p = (char *) record; + char *pe; + + /* set pe to the beginning of the next page */ + pe = (char *) record + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +880,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +990,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..8e1fa32489 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,10 +471,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", + startpointTLI, + LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..3eeba220a1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..67d264df26 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,101 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +my $chkptfile; +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# and the end-of-wal messages shouldn't be seen +# the same message has been confirmed in the past +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Tue_Feb__1_11_58_01_2022_779)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v11] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 93 +++++++++++++----- src/backend/access/transam/xlogreader.c | 78 +++++++++++++++ src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 110 +++++++++++++++++++++- 6 files changed, 271 insertions(+), 31 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 958220c495..618f33d342 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-WAL, emit the error + * message right now. Otherwise the message if any is shown as a + * part of the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4553,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4571,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4629,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (xlogreader->EndOfWAL) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7294,7 +7334,7 @@ StartupXLOG(void) { ereport(LOG, (errmsg("database system was not properly shut down; " - "automatic recovery in progress"))); + "crash recovery in progress"))); if (recoveryTargetTLI > ControlFile->checkPointCopy.ThisTimeLineID) ereport(LOG, (errmsg("crash recovery starts in timeline %u " @@ -7544,7 +7584,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7822,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7882,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13097,7 +13144,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..03a8b42f15 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,40 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p; + char *pe; + + /* scan from the beginning of the record to the end of block */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +881,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +991,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..1a7a692bc0 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,10 +471,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..3eeba220a1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..67d264df26 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,101 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +my $chkptfile; +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# and the end-of-wal messages shouldn't be seen +# the same message has been confirmed in the past +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Wed_Feb__9_16_44_14_2022_277)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v16] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 144 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 92 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++ 6 files changed, 305 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..bd0f211a23 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -42,6 +42,8 @@ static bool allocate_recordbuf(XLogReaderState *state, uint32 reclength); static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -121,6 +123,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +295,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -371,25 +375,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->ReadRecPtr, record, @@ -399,18 +399,13 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ; if (total_len > len) { @@ -588,6 +583,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -719,6 +723,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -730,14 +788,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (record->xl_rmid > RM_MAX_ID) { report_invalid_record(state, @@ -836,6 +889,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +999,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index f9f212680b..750056acaf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1592,7 +1592,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -1706,7 +1706,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1765,13 +1765,20 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that indicate @@ -2939,6 +2946,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -2954,6 +2962,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2963,13 +2983,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -3000,11 +3023,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3017,11 +3043,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3042,12 +3073,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3129,12 +3172,16 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return -1; } } @@ -3767,7 +3814,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ceaff097b9..4f117ea4da 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 2340dc247b..215abe95dc 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1173,9 +1173,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..7b314ef10e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 14154d1ce0..01033334d6 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,7 +10,9 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -48,7 +50,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -61,4 +71,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Fri_Mar__4_09_43_59_2022_493)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v13] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 91 ++++++++++++++----- src/backend/access/transam/xlogreader.c | 78 ++++++++++++++++ src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++++++++ 6 files changed, 266 insertions(+), 30 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 958220c495..bb7026ac77 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4557,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4577,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4634,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7580,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7818,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7878,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -12434,12 +12477,14 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; - + xlogreader->EndOfWAL = true; return -1; } } @@ -13097,7 +13142,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..22982c4de7 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,40 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p; + char *pe; + + /* scan from the beginning of the record to the end of block */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +881,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +991,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..1a7a692bc0 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,10 +471,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..7b314ef10e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 14154d1ce0..01033334d6 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,7 +10,9 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -48,7 +50,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -61,4 +71,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Tue_Feb_15_11_22_38_2022_332)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v5] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 94 +++++++++++++++++++------ src/backend/access/transam/xlogreader.c | 14 ++++ src/backend/replication/walreceiver.c | 3 +- src/include/access/xlogreader.h | 1 + 4 files changed, 87 insertions(+), 25 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 5cda30836f..623fb01d0a 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4477,6 +4477,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); ReadRecPtr = xlogreader->ReadRecPtr; @@ -4494,6 +4495,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = ReadRecPtr ? ReadRecPtr : EndRecPtr; } if (readFile >= 0) @@ -4503,13 +4514,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error message + * right now. Otherwise the message if any is shown as a part of + * the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4540,11 +4550,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4557,11 +4568,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4609,12 +4626,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7582,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, ThisTimeLineID); + record = ReadRecord(xlogreader, WARNING, false, ThisTimeLineID); } if (record != NULL) @@ -7781,7 +7819,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, ThisTimeLineID); + record = ReadRecord(xlogreader, WARNING, false, ThisTimeLineID); } while (record != NULL); /* @@ -7841,13 +7879,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13135,7 +13180,9 @@ XLogShutdownWalRcv(void) * reading from pg_wal, because we don't expect any invalid records in archive * or in records streamed from the primary. Files in the archive should be complete, * and we should never hit the end of WAL because we stop and wait for more WAL - * to arrive before replaying it. + * to arrive before replaying it. When we failed to read a new page, + * readSource is reset to XLOG_FROM_ANY. This indicates all sources including + * pg_wal was failed. Thus treat that the same way with XLOG_FROM_PG_WAL. * * NOTE: This function remembers the RecPtr value it was last called with, * to suppress repeated messages about the same record. Only call this when @@ -13147,7 +13194,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if ((readSource == XLOG_FROM_PG_WAL || readSource == XLOG_FROM_ANY) + && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index f39f8044a9..df2198e862 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,9 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +735,15 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* This is strictly not an invalid state, so phrase it as so. */ + report_invalid_record(state, + "record length is 0 at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + state->EndOfWAL = true; + return false; + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7a7eb3784e..ba3c4bd550 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,8 +471,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index de6fd791fe..1241b85838 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the -- 2.27.0 ----Next_Part(Mon_Nov__8_14_59_46_2021_800)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v17] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 145 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 92 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++ 6 files changed, 306 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index e437c42992..0942265408 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -46,6 +46,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool non_blocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -147,6 +149,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -552,6 +555,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -633,25 +637,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -661,18 +661,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -904,6 +900,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } if (decoded && decoded->oversized) @@ -1083,6 +1088,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1094,14 +1153,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (record->xl_rmid > RM_MAX_ID) { report_invalid_record(state, @@ -1200,6 +1254,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1285,6 +1364,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 9feea3e6ec..98382d66a4 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1592,7 +1592,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -1706,7 +1706,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1765,13 +1765,20 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that indicate @@ -2939,6 +2946,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -2954,6 +2962,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2963,13 +2983,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -3000,11 +3023,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3017,11 +3043,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3042,12 +3073,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3129,12 +3172,16 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return -1; } } @@ -3767,7 +3814,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ceaff097b9..4f117ea4da 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index fc081adfb8..9bebca8154 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1174,9 +1174,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index f4388cc9be..21a8f9552c 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -201,6 +201,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 14154d1ce0..01033334d6 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,7 +10,9 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -48,7 +50,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -61,4 +71,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Tue_Mar_22_11_34_46_2022_933)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v11] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 91 +++++++++++++----- src/backend/access/transam/xlogreader.c | 78 +++++++++++++++ src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 110 +++++++++++++++++++++- 6 files changed, 270 insertions(+), 30 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index dfe2a0bcce..378c13ccf7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at the + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = + xlogreader->ReadRecPtr ? + xlogreader->ReadRecPtr : xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error + * message right now. Otherwise the message if any is shown as a + * part of the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4553,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4571,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4629,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7584,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7822,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7882,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13097,7 +13144,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..03a8b42f15 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,40 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + char *p; + char *pe; + + /* scan from the beginning of the record to the end of block */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid record length */ + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +881,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -921,6 +991,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..8e1fa32489 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,10 +471,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", + startpointTLI, + LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..3eeba220a1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..67d264df26 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,101 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +my $chkptfile; +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# and the end-of-wal messages shouldn't be seen +# the same message has been confirmed in the past +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Wed_Feb__2_14_34_58_2022_606)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v8] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 91 +++++++++++++----- src/backend/access/transam/xlogreader.c | 64 +++++++++++++ src/backend/replication/walreceiver.c | 3 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 110 +++++++++++++++++++++- 6 files changed, 254 insertions(+), 28 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 58922f7ede..c08b9554b3 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4480,6 +4480,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -4495,6 +4496,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at the + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = + xlogreader->ReadRecPtr ? + xlogreader->ReadRecPtr : xlogreader->EndRecPtr; } if (readFile >= 0) @@ -4504,13 +4517,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error message + * right now. Otherwise the message if any is shown as a part of + * the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4541,11 +4553,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4558,11 +4571,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4610,12 +4629,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7584,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -7782,7 +7822,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -7842,13 +7882,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13097,7 +13144,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 35029cf97d..55f54cd98d 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +741,36 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the whole + * header is zeroed. + */ + char *p = (char *)record; + char *pe = (char *)record + SizeOfXLogRecord; + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* it is completely zeroed, call it a day */ + report_invalid_record(state, "empty record header found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + } + else + { + /* Otherwise the header is corrupted. */ + report_invalid_record(state, "garbage record header at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + } + + return false; + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -836,6 +877,29 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0 ; i < XLOG_BLCKSZ && phdr[i] == 0 ; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index b39fce8c23..3034f8281e 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,8 +471,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a6251e1a96..3745e76488 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1176,9 +1176,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 477f0efe26..3eeba220a1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 3892aba3e5..b793280a5c 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -10,9 +10,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; use Config; +use IPC::Run; -plan tests => 3; +plan tests => 11; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -50,7 +52,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -62,3 +72,101 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +my $chkptfile; +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: garbage record header at 0/$lastlsn", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# and the end-of-wal messages shouldn't be seen +# the same message has been confirmed in the past +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Tue_Jan_25_17_34_56_2022_438)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v6] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlog.c | 89 +++++++++++++++++++------ src/backend/access/transam/xlogreader.c | 42 ++++++++++++ src/backend/replication/walreceiver.c | 3 +- src/include/access/xlogreader.h | 1 + 4 files changed, 111 insertions(+), 24 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 5cda30836f..e90c69810b 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4477,6 +4477,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); ReadRecPtr = xlogreader->ReadRecPtr; @@ -4494,6 +4495,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * NULL ReadRecPtr means we could not read a record at + * beginning. In that case EndRecPtr is storing the LSN of the + * record we tried to read. + */ + ErrRecPtr = ReadRecPtr ? ReadRecPtr : EndRecPtr; } if (readFile >= 0) @@ -4503,13 +4514,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we get here for other than end-of-wal, emit the error message + * right now. Otherwise the message if any is shown as a part of + * the end-of-WAL message below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4540,11 +4550,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4557,11 +4568,17 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4609,12 +4626,33 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * If we haven't emit an error message, we have safely reached the + * end-of-WAL. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, LSN_FORMAT_ARGS(ErrRecPtr), replayTLI, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -7544,7 +7582,7 @@ StartupXLOG(void) else { /* just have to read next record after CheckPoint */ - record = ReadRecord(xlogreader, LOG, false, ThisTimeLineID); + record = ReadRecord(xlogreader, WARNING, false, ThisTimeLineID); } if (record != NULL) @@ -7781,7 +7819,7 @@ StartupXLOG(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, ThisTimeLineID); + record = ReadRecord(xlogreader, WARNING, false, ThisTimeLineID); } while (record != NULL); /* @@ -7841,13 +7879,20 @@ StartupXLOG(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that @@ -13147,7 +13192,7 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index f39f8044a9..273b927cd9 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -121,6 +121,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -292,6 +293,7 @@ XLogReadRecord(XLogReaderState *state, char **errormsg) /* reset error state */ *errormsg = NULL; state->errormsg_buf[0] = '\0'; + state->EndOfWAL = false; ResetDecoder(state); state->abortedRecPtr = InvalidXLogRecPtr; @@ -588,6 +590,16 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the messages is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message at + * it should be more detailed. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } /* @@ -730,6 +742,36 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + if (record->xl_tot_len == 0) + { + /* + * We are almost sure reaching the end of WAL, make sure that the whole + * header is zeroed. + */ + char *p = (char *)record; + char *pe = (char *)record + SizeOfXLogRecord; + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* it is completely zeroed, call it a day */ + report_invalid_record(state, "empty record header found at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + } + else + { + /* Otherwise we found a garbage header.. */ + report_invalid_record(state, "garbage record header at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + } + + return false; + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7a7eb3784e..ba3c4bd550 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -471,8 +471,7 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, LSN_FORMAT_ARGS(LogstreamResult.Write)))); endofwal = true; diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index de6fd791fe..1241b85838 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -174,6 +174,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* the last attempt was EOW? */ /* * Set at the end of recovery: the start point of a partial record at the -- 2.27.0 ----Next_Part(Tue_Nov__9_16_27_52_2021_065)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v3] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Make this message less scary as "reached end of WAL". --- src/backend/access/transam/xlog.c | 69 ++++++++++++++++++------- src/backend/access/transam/xlogreader.c | 9 ++++ src/backend/replication/walreceiver.c | 11 ++-- 3 files changed, 67 insertions(+), 22 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 793c076da6..6c2924dfb7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -4283,12 +4283,15 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); ReadRecPtr = xlogreader->ReadRecPtr; EndRecPtr = xlogreader->EndRecPtr; if (record == NULL) { + ErrRecPtr = ReadRecPtr ? ReadRecPtr : EndRecPtr; + if (readFile >= 0) { close(readFile); @@ -4296,13 +4299,13 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * If we are fetching checkpoint, we emit the error message right + * now. Otherwise the error is regarded as "end of WAL" and the + * message if any is shown as a part of the end-of-WAL message + * below. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, EndRecPtr), + if (fetching_ckpt && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -4333,11 +4336,12 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -4350,11 +4354,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in %s during crash recovery, entering archive recovery", + (uint32) (ErrRecPtr >> 32), + (uint32) ErrRecPtr, + ThisTimeLineID, + xlogSourceNames[currentSource]))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -4392,12 +4403,34 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * We reached the end of WAL, show the messages just once at the + * same LSN. + */ + if (emode_for_corrupt_record(LOG, ErrRecPtr) == LOG) + { + char *fmt; + + if (StandbyMode) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during standby mode"); + else if (InArchiveRecovery) + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during archive recovery"); + else + fmt = gettext_noop("reached end of WAL at %X/%X on timeline %u in %s during crash recovery"); + + ereport(LOG, + (errmsg(fmt, (uint32) (EndRecPtr >> 32), + (uint32) EndRecPtr, ThisTimeLineID, + xlogSourceNames[currentSource]), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); + } } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 32f02256ed..9ea1305364 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -682,6 +682,15 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { + + if (record->xl_tot_len == 0) + { + /* This is strictly not an invalid state, so phrase it as so. */ + report_invalid_record(state, + "record length is 0 at %X/%X", + (uint32) (RecPtr >> 32), (uint32) RecPtr); + return false; + } if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 25e0333c9e..da978d4047 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -479,12 +479,15 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", + (errmsg("replication terminated by primary server on timeline %u at %X/%X.", startpointTLI, (uint32) (LogstreamResult.Write >> 32), (uint32) LogstreamResult.Write))); - endofwal = true; - break; + + /* + * we have no longer anything to do on the broken + * connection other than exiting. + */ + proc_exit(1); } len = walrcv_receive(wrconn, &buf, &wait_fd); } -- 2.18.2 ----Next_Part(Tue_Mar_24_10_52_50_2020_211)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v18] Make End-Of-Recovery error less scary @ 2020-02-28 06:52 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2020-02-28 06:52 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 145 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 92 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++ 6 files changed, 306 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index e437c42992..0942265408 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -46,6 +46,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool non_blocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -147,6 +149,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -552,6 +555,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -633,25 +637,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -661,18 +661,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -904,6 +900,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } if (decoded && decoded->oversized) @@ -1083,6 +1088,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1094,14 +1153,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (record->xl_rmid > RM_MAX_ID) { report_invalid_record(state, @@ -1200,6 +1254,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1285,6 +1364,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 8b22c4e634..de8be3b834 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1592,7 +1592,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } if (record != NULL) @@ -1706,7 +1706,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogreader, LOG, false, replayTLI); + record = ReadRecord(xlogreader, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1765,13 +1765,20 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + (errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database."))); + } /* * This check is intentionally after the above log messages that indicate @@ -2949,6 +2956,7 @@ ReadRecord(XLogReaderState *xlogreader, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogReadRecord(xlogreader, &errormsg); if (record == NULL) @@ -2964,6 +2972,18 @@ ReadRecord(XLogReaderState *xlogreader, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2973,13 +2993,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, } /* - * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * We only end up here without a message when XLogPageRead() failed + * in that case we already logged something, or just met end-of-WAL + * conditions. In StandbyMode that only happens if we have been + * triggered, so we shouldn't loop anymore in that case. When + * EndOfWAL is true, we don't emit that error if any immediately + * and instead will show it as a part of a decent end-of-wal + * message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), (errmsg_internal("%s", errormsg) /* already translated */ )); } @@ -3010,11 +3033,14 @@ ReadRecord(XLogReaderState *xlogreader, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3027,11 +3053,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + (errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI))); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3052,12 +3083,24 @@ ReadRecord(XLogReaderState *xlogreader, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + (errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0))); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3139,12 +3182,16 @@ retry: private->replayTLI, xlogreader->EndRecPtr)) { + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return -1; } } @@ -3777,7 +3824,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 3c9411e221..2f9ef9bf31 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + (errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI))); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 4cb40d068a..2a78e954de 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1310,9 +1310,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - fatal_error("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + fatal_error("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index f4388cc9be..21a8f9552c 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -201,6 +201,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..f8b4a8417c 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway break the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /fatal: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.27.0 ----Next_Part(Tue_Mar_29_15_07_01_2022_552)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v19] Make End-Of-Recovery error less scary @ 2022-07-07 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-07-07 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 145 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 95 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 ++++++++++++++++ 6 files changed, 308 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index f3dc4b7797..a86ab2b02b 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool non_blocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -554,6 +557,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -635,25 +639,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -663,18 +663,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -906,6 +902,15 @@ err: */ state->abortedRecPtr = RecPtr; state->missingContrecPtr = targetPagePtr; + + /* + * If the message is not set yet, that means we failed to load the + * page for the record. Otherwise do not hide the existing message. + */ + if (state->errormsg_buf[0] == '\0') + report_invalid_record(state, + "missing contrecord at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); } if (decoded && decoded->oversized) @@ -1085,6 +1090,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1096,14 +1155,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1202,6 +1256,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1287,6 +1366,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 5d6f1b5e46..baae4e84cf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1622,7 +1622,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1731,7 +1731,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1785,11 +1785,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -2969,6 +2977,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -2984,6 +2993,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -2994,13 +3015,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3030,11 +3053,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3047,11 +3073,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3072,12 +3103,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3172,11 +3215,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3830,7 +3878,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 3d37c1fe62..73f7641c4f 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 6528113628..bd09a62a9d 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 5395f155aa..b2dae53557 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..bde16b7cfa 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Thu_Jul__7_17_32_33_2022_801)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Make End-Of-Recovery error less scary @ 2022-07-07 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-07-07 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 135 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 95 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 299 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b2544..f891a629443 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1106,16 +1102,47 @@ XLogReaderInvalReadState(XLogReaderState *state) } /* - * Validate an XLOG record header. + * Validate record length of an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in - * XLogReadRecord. It's not intended for use from anywhere else. + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. */ static bool -ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, - XLogRecPtr PrevRecPtr, XLogRecord *record, - bool randAccess) +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) { + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -1124,6 +1151,24 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, (uint32) SizeOfXLogRecord, record->xl_tot_len); return false; } + + return true; +} + +/* + * Validate an XLOG record header. + * + * This is just a convenience subroutine to avoid duplicated code in + * XLogReadRecord. It's not intended for use from anywhere else. + */ +static bool +ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecPtr PrevRecPtr, XLogRecord *record, + bool randAccess) +{ + if (!ValidXLogRecordLength(state, RecPtr, record)) + return false; + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cb07694aea6..0034f65af02 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3032,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3054,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3076,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3114,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3134,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3164,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3276,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3946,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 6cbb67c92a3..7ef19e435ce 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca58..26a4125b301 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316ae..70d3b25edaf 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046d..bde16b7cfa7 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.25.1 --nHJAUhyIZkPvF01C-- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Make End-Of-Recovery error less scary @ 2022-07-07 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-07-07 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 134 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 95 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 298 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 050d2f424e4..9b8f29d0ad0 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool non_blocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1106,16 +1102,47 @@ XLogReaderInvalReadState(XLogReaderState *state) } /* - * Validate an XLOG record header. + * Validate record length of an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in - * XLogReadRecord. It's not intended for use from anywhere else. + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. */ static bool -ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, - XLogRecPtr PrevRecPtr, XLogRecord *record, - bool randAccess) +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) { + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -1124,6 +1151,24 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, (uint32) SizeOfXLogRecord, record->xl_tot_len); return false; } + + return true; +} + +/* + * Validate an XLOG record header. + * + * This is just a convenience subroutine to avoid duplicated code in + * XLogReadRecord. It's not intended for use from anywhere else. + */ +static bool +ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecPtr PrevRecPtr, XLogRecord *record, + bool randAccess) +{ + if (!ValidXLogRecordLength(state, RecPtr, record)) + return false; + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1222,6 +1267,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1307,6 +1377,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index b41e6826643..caa3b5e5b31 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3032,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3054,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3076,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3114,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3134,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3164,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3276,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3946,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index f6ef0ace2c4..c6d7be66885 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca58..26a4125b301 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 6afec33d418..264afb6a78e 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046d..bde16b7cfa7 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.25.1 --sMP3l7aO1Ldl3x+Z-- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v21] Make End-Of-Recovery error less scary @ 2022-07-07 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-07-07 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 136 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 298 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 4d6c34e0fc..b03eeb1487 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,6 +1101,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1116,14 +1166,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1222,6 +1267,31 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLogSegNoOffsetToRecPtr(segno, offset, state->segcxt.ws_segsize, recaddr); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1307,6 +1377,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recaddr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index b41e682664..56f29e73fe 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3031,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3053,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3075,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3113,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3133,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3163,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3275,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3945,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index f6ef0ace2c..c6d7be6688 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -472,10 +472,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca5..26a4125b30 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 6dcde2523a..0818cb7ef0 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..bde16b7cfa 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Mon_Sep_26_16_17_37_2022_881)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v22] Make End-Of-Recovery error less scary @ 2022-11-15 04:41 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-15 04:41 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 137 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 299 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b254..f891a62944 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,6 +1101,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1116,14 +1166,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cb07694aea..3f54c875e5 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3031,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3053,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3075,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3113,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3133,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3163,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3275,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3945,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa..054a4cb127 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca5..26a4125b30 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316a..70d3b25eda 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..bde16b7cfa 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Fri_Nov_18_17_25_37_2022_798)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v22] Make End-Of-Recovery error less scary @ 2022-11-15 04:41 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-15 04:41 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 137 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 299 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b254..f891a62944 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,6 +1101,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (*p == 0 && p < pe) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1116,14 +1166,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cb07694aea..3f54c875e5 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3031,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3053,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3075,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3113,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3133,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3163,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3275,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3945,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa..054a4cb127 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca5..26a4125b30 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316a..70d3b25eda 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..bde16b7cfa 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Fri_Nov_18_17_25_37_2022_798)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Make End-Of-Recovery error less scary @ 2022-11-15 04:41 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-15 04:41 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 135 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 298 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b2544..137de967951 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1106,16 +1102,47 @@ XLogReaderInvalReadState(XLogReaderState *state) } /* - * Validate an XLOG record header. + * Validate record length of an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in - * XLogReadRecord. It's not intended for use from anywhere else. + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. */ static bool -ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, - XLogRecPtr PrevRecPtr, XLogRecord *record, - bool randAccess) +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) { + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -1124,6 +1151,24 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, (uint32) SizeOfXLogRecord, record->xl_tot_len); return false; } + + return true; +} + +/* + * Validate an XLOG record header. + * + * This is just a convenience subroutine to avoid duplicated code in + * XLogReadRecord. It's not intended for use from anywhere else. + */ +static bool +ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecPtr PrevRecPtr, XLogRecord *record, + bool randAccess) +{ + if (!ValidXLogRecordLength(state, RecPtr, record)) + return false; + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cb07694aea6..3f54c875e5a 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3031,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3053,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3075,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3113,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3133,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3163,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3275,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3945,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa6..054a4cb127a 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca58..26a4125b301 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316ae..70d3b25edaf 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046d..bde16b7cfa7 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.25.1 --aYDVKSzuImP48n7V-- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Make End-Of-Recovery error less scary @ 2022-11-15 04:41 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-15 04:41 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 135 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 106 +++++++++++++++++ 6 files changed, 298 insertions(+), 58 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b2544..137de967951 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1106,16 +1102,47 @@ XLogReaderInvalReadState(XLogReaderState *state) } /* - * Validate an XLOG record header. + * Validate record length of an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in - * XLogReadRecord. It's not intended for use from anywhere else. + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. */ static bool -ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, - XLogRecPtr PrevRecPtr, XLogRecord *record, - bool randAccess) +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) { + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + if (record->xl_tot_len < SizeOfXLogRecord) { report_invalid_record(state, @@ -1124,6 +1151,24 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, (uint32) SizeOfXLogRecord, record->xl_tot_len); return false; } + + return true; +} + +/* + * Validate an XLOG record header. + * + * This is just a convenience subroutine to avoid duplicated code in + * XLogReadRecord. It's not intended for use from anywhere else. + */ +static bool +ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecPtr PrevRecPtr, XLogRecord *record, + bool randAccess) +{ + if (!ValidXLogRecordLength(state, RecPtr, record)) + return false; + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cb07694aea6..3f54c875e5a 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1626,7 +1626,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1735,7 +1735,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1789,11 +1789,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3024,6 +3031,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3045,6 +3053,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3055,13 +3075,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3091,11 +3113,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else - { - /* No valid record available from this source */ - lastSourceFailed = true; + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) + { /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3108,11 +3133,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3133,12 +3163,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3233,11 +3275,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3898,7 +3945,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa6..054a4cb127a 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca58..26a4125b301 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316ae..70d3b25edaf 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046d..bde16b7cfa7 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,15 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +my $max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, $reached_eow_pat, $logstart)); + sleep 0.5; +} +ok ($max_attempts >= 0, "end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +70,100 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$max_attempts = 360; +while ($max_attempts-- >= 0) +{ + last if (find_in_log($node, "WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart)); + sleep 0.5; +} +ok($max_attempts >= 0, "header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.25.1 --aYDVKSzuImP48n7V-- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v23] Make End-Of-Recovery error less scary @ 2022-11-30 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-30 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 137 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 96 +++++++++++++++ 6 files changed, 289 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b254..137de96795 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,6 +1101,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1116,14 +1166,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 41ffc57da9..049b468620 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1625,7 +1625,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1734,7 +1734,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1788,11 +1788,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3020,6 +3027,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3041,6 +3049,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3051,13 +3071,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3087,11 +3109,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3104,11 +3129,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3129,12 +3159,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3229,11 +3271,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3899,7 +3946,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa..054a4cb127 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca5..26a4125b30 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316a..70d3b25eda 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..8a379e8988 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,10 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +$node->wait_for_log($reached_eow_pat, $logstart); +pass("end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +65,95 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$node->wait_for_log("WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart); +pass("header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Wed_Nov_30_11_56_02_2022_262)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v23] Make End-Of-Recovery error less scary @ 2022-11-30 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-30 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. --- src/backend/access/transam/xlogreader.c | 137 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/011_crash_recovery.pl | 96 +++++++++++++++ 6 files changed, 289 insertions(+), 59 deletions(-) diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 93f667b254..137de96795 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,6 +1101,60 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * @@ -1116,14 +1166,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1264,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1304,6 +1375,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, LSN_FORMAT_ARGS(hdr->xlp_pageaddr), fname, offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 41ffc57da9..049b468620 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1625,7 +1625,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1734,7 +1734,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1788,11 +1788,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3020,6 +3027,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3041,6 +3049,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3051,13 +3071,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3087,11 +3109,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3104,11 +3129,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) StandbyMode = true; @@ -3129,12 +3159,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3229,11 +3271,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3899,7 +3946,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index ad383dbcaa..054a4cb127 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca5..26a4125b30 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1168,9 +1168,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e87f91316a..70d3b25eda 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl index 1b57d01046..8a379e8988 100644 --- a/src/test/recovery/t/011_crash_recovery.pl +++ b/src/test/recovery/t/011_crash_recovery.pl @@ -9,7 +9,9 @@ use warnings; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use IPC::Run; +my $reached_eow_pat = "reached end of WAL at "; my $node = PostgreSQL::Test::Cluster->new('primary'); $node->init(allows_streaming => 1); $node->start; @@ -47,7 +49,10 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), # Crash and restart the postmaster $node->stop('immediate'); +my $logstart = get_log_size($node); $node->start; +$node->wait_for_log($reached_eow_pat, $logstart); +pass("end-of-wal is logged"); # Make sure we really got a new xid cmp_ok($node->safe_psql('postgres', 'SELECT pg_current_xact_id()'), @@ -60,4 +65,95 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]), $stdin .= "\\q\n"; $tx->finish; # wait for psql to quit gracefully +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # halfway corrupt the last record +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains +$logstart = get_log_size($node); +$node->start; +$node->wait_for_log("WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart); +pass("header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +$node->stop('immediate'); + done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Wed_Nov_30_11_56_02_2022_262)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v24] Make End-Of-Recovery error less scary @ 2022-11-30 02:51 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2022-11-30 02:51 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 144 +++++++++++++++++----- src/backend/access/transam/xlogrecovery.c | 94 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/034_recovery.pl | 135 ++++++++++++++++++++ 6 files changed, 335 insertions(+), 59 deletions(-) create mode 100644 src/test/recovery/t/034_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index aa6c929477..8cb2d55333 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -640,25 +644,21 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Validate the record header. * - * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. - */ - record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; - - /* - * If the whole record header is on this page, validate it immediately. - * Otherwise do just a basic sanity check on xl_tot_len, and validate the - * rest of the header after reading it from the next page. The xl_tot_len + * Even though we use an XLogRecord pointer here, the whole record header + * might not fit on this page. If the whole record header is on this page, + * validate it immediately. Even otherwise xl_tot_len must be on this page + * (it is the first field of MAXALIGNed records), but we still cannot + * access any further fields until we've verified that we got the whole + * header, so do just a basic sanity check on record length, and validate + * the rest of the header after reading it from the next page. The length * check is necessary here to ensure that we enter the "Need to reassemble * record" code path below; otherwise we might fail to apply * ValidXLogRecordHeader at all. */ + record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); + if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord) { if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record, @@ -668,18 +668,14 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1105,25 +1101,81 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: wanted %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * * This is just a convenience subroutine to avoid duplicated code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: wanted %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1219,6 +1271,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1308,6 +1386,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index dbe9394762..3ada8b346d 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1644,7 +1644,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1753,7 +1753,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1807,11 +1807,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3044,6 +3051,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3065,6 +3073,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3075,13 +3095,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3112,11 +3134,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3129,11 +3154,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3154,12 +3184,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3256,11 +3298,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3928,7 +3975,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index f6446da2d6..78b85c5a25 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 44b5c8726e..0f0cccd425 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1276,9 +1276,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index d77bb2ab9b..84562d9de1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/034_recovery.pl b/src/test/recovery/t/034_recovery.pl new file mode 100644 index 0000000000..580ae3b9f1 --- /dev/null +++ b/src/test/recovery/t/034_recovery.pl @@ -0,0 +1,135 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = get_log_size($node); +$node->start; +$node->wait_for_log("WARNING: invalid record length at 0/$lastlsn: wanted [0-9]+, got 0", + $logstart); +pass("header error is logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = get_log_size($node); +$node->start(); +$node->wait_for_log($reached_eow_pat, $logstart); + +# restart the standby +$logstart = get_log_size($node_standby); +$node_standby->start(); +$node_standby->wait_for_log($reached_eow_pat, $logstart); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# return the size of logfile of $node in bytes +sub get_log_size +{ + my ($node) = @_; + + return (stat $node->logfile)[7]; +} + +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Tue_Feb__7_16_07_03_2023_615)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v28] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 7 files changed, 380 insertions(+), 61 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index e00395ff2b..fd6b1bf7e1 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Thu_Jan_11_16_18_16_2024_827)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v28] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 7 files changed, 380 insertions(+), 61 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index e00395ff2b..fd6b1bf7e1 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Thu_Jan_11_16_18_16_2024_827)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v25] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 135 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++++ 6 files changed, 329 insertions(+), 51 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index cadea21b37..5a27c10bbb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -558,6 +561,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -641,16 +645,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -669,18 +669,21 @@ restart: } else { - /* XXX: more validation should be done here */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + * + * XXX: more validation should be done here + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } + gotheader = false; } + total_len = record->xl_tot_len; + /* * Find space to decode this record. Don't allow oversized allocation if * the caller requested nonblocking. Otherwise, we *have* to try to @@ -1106,25 +1109,81 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * * This is just a convenience subroutine to avoid duplicated code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1220,6 +1279,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1309,6 +1394,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index dbe9394762..3ada8b346d 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1644,7 +1644,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1753,7 +1753,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1807,11 +1807,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3044,6 +3051,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3065,6 +3073,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3075,13 +3095,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3112,11 +3134,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3129,11 +3154,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3154,12 +3184,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3256,11 +3298,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3928,7 +3975,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index f6446da2d6..78b85c5a25 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -498,10 +498,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 44b5c8726e..0f0cccd425 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1276,9 +1276,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index d77bb2ab9b..84562d9de1 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.31.1 ----Next_Part(Tue_Mar__7_15_35_47_2023_595)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v27] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 134 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++++ 6 files changed, 327 insertions(+), 52 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index e0baa86bd3..ce65f99c60 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -525,6 +528,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -608,16 +612,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -636,19 +636,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1091,25 +1091,81 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * * This is just a convenience subroutine to avoid duplicated code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1207,6 +1263,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1296,6 +1378,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c61566666a..9db27b765f 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1672,7 +1672,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1781,7 +1781,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1835,11 +1835,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3091,6 +3098,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3112,6 +3120,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3122,13 +3142,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3159,11 +3181,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3176,11 +3201,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3201,12 +3231,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3303,11 +3345,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3975,7 +4022,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 2398167f49..2271304814 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -496,10 +496,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a3535bdfa9..11a69027d0 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 0813722715..2be8ea7c37 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.39.3 ----Next_Part(Wed_Nov_22_16_31_10_2023_079)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v29] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/bin/pg_waldump/t/001_basic.pl | 5 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 8 files changed, 383 insertions(+), 63 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 728059518e..c69e33dbe7 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index 082cb8a589..b15e7a2ca3 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -147,10 +147,11 @@ command_fails_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile command_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile, $node->data_dir . '/pg_wal/' . $end_walfile ], qr/./, 'runs with start and end segment specified'); command_fails_like([ 'pg_waldump', '-p', $node->data_dir ], qr/error: no start WAL location given/, 'path option requires start location'); command_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn, '--end', $end_lsn ], qr/./, 'runs with path option and start and end locations'); -command_fails_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'falling off the end of the WAL results in an error'); + +command_checks_all([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/./], [qr/pg_waldump: end of WAL at.*: empty record at/], 'the end of the WAL doesn\'t result in an error'); command_like([ 'pg_waldump', '--quiet', $node->data_dir . '/pg_wal/' . $start_walfile ], qr/^$/, 'no output with --quiet option'); -command_fails_like([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'errors are shown with --quiet'); +command_checks_all([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/^$/], [qr/pg_waldump: end of WAL at .*: empty record at .*/], 'end-status is shown with --quiet'); # Test for: Display a message that we're skipping data if `from` diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Tue_Jan_23_13_13_56_2024_752)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v27] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 134 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 94 +++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 ++- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++++ 6 files changed, 327 insertions(+), 52 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index e0baa86bd3..ce65f99c60 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -525,6 +528,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -608,16 +612,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -636,19 +636,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1091,25 +1091,81 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * The page after the record is completely zeroed. That suggests + * we don't have a record after this point. We don't bother + * checking the pages after since they are not zeroed in the case + * of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * * This is just a convenience subroutine to avoid duplicated code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1207,6 +1263,32 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + report_invalid_record(state, + "empty page in log segment %s, offset %u", + fname, + offset); + state->EndOfWAL = true; + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1296,6 +1378,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c61566666a..9db27b765f 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1672,7 +1672,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1781,7 +1781,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1835,11 +1835,18 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, - (errmsg("redo is not required"))); + errmsg("redo is not required")); + } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); } /* @@ -3091,6 +3098,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3112,6 +3120,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3122,13 +3142,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3159,11 +3181,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3176,11 +3201,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3201,12 +3231,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3303,11 +3345,16 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* promotion exit is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3975,7 +4022,8 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* use currentSource as readSource is reset at failure */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 2398167f49..2271304814 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -496,10 +496,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index a3535bdfa9..11a69027d0 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 0813722715..2be8ea7c37 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} -- 2.39.3 ----Next_Part(Wed_Nov_22_16_31_10_2023_079)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v28] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 7 files changed, 380 insertions(+), 61 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index e00395ff2b..fd6b1bf7e1 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Thu_Jan_11_16_18_16_2024_827)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v29] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/bin/pg_waldump/t/001_basic.pl | 5 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 8 files changed, 383 insertions(+), 63 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 728059518e..c69e33dbe7 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index 082cb8a589..b15e7a2ca3 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -147,10 +147,11 @@ command_fails_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile command_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile, $node->data_dir . '/pg_wal/' . $end_walfile ], qr/./, 'runs with start and end segment specified'); command_fails_like([ 'pg_waldump', '-p', $node->data_dir ], qr/error: no start WAL location given/, 'path option requires start location'); command_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn, '--end', $end_lsn ], qr/./, 'runs with path option and start and end locations'); -command_fails_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'falling off the end of the WAL results in an error'); + +command_checks_all([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/./], [qr/pg_waldump: end of WAL at.*: empty record at/], 'the end of the WAL doesn\'t result in an error'); command_like([ 'pg_waldump', '--quiet', $node->data_dir . '/pg_wal/' . $start_walfile ], qr/^$/, 'no output with --quiet option'); -command_fails_like([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'errors are shown with --quiet'); +command_checks_all([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/^$/], [qr/pg_waldump: end of WAL at .*: empty record at .*/], 'end-status is shown with --quiet'); # Test for: Display a message that we're skipping data if `from` diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Tue_Jan_23_13_13_56_2024_752)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH v29] Make End-Of-Recovery error less scary @ 2023-03-07 05:55 Kyotaro Horiguchi <horikyota.ntt@gmail.com> 0 siblings, 0 replies; 39+ messages in thread From: Kyotaro Horiguchi @ 2023-03-07 05:55 UTC (permalink / raw) When recovery in any type ends, we see a bit scary error message like "invalid record length" that suggests something serious is happening. Actually if recovery meets a record with length = 0, that usually means it finished applying all available WAL records. Make this message less scary as "reached end of WAL". Instead, raise the error level for other kind of WAL failure to WARNING. To make sure that the detection is correct, this patch checks if all trailing bytes in the same page are zeroed in that case. --- src/backend/access/transam/xlogreader.c | 147 ++++++++++++++++++---- src/backend/access/transam/xlogrecovery.c | 96 ++++++++++---- src/backend/replication/walreceiver.c | 7 +- src/bin/pg_waldump/pg_waldump.c | 13 +- src/bin/pg_waldump/t/001_basic.pl | 5 +- src/include/access/xlogreader.h | 1 + src/test/recovery/t/035_recovery.pl | 130 +++++++++++++++++++ src/test/recovery/t/039_end_of_wal.pl | 47 +++++-- 8 files changed, 383 insertions(+), 63 deletions(-) create mode 100644 src/test/recovery/t/035_recovery.pl diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7190156f2f..94861969eb 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -48,6 +48,8 @@ static int ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen); static void XLogReaderInvalReadState(XLogReaderState *state); static XLogPageReadResult XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking); +static bool ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record); static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess); static bool ValidXLogRecord(XLogReaderState *state, XLogRecord *record, @@ -149,6 +151,7 @@ XLogReaderAllocate(int wal_segment_size, const char *waldir, pfree(state); return NULL; } + state->EndOfWAL = false; state->errormsg_buf[0] = '\0'; /* @@ -553,6 +556,7 @@ XLogDecodeNextRecord(XLogReaderState *state, bool nonblocking) /* reset error state */ state->errormsg_buf[0] = '\0'; decoded = NULL; + state->EndOfWAL = false; state->abortedRecPtr = InvalidXLogRecPtr; state->missingContrecPtr = InvalidXLogRecPtr; @@ -636,16 +640,12 @@ restart: Assert(pageHeaderSize <= readOff); /* - * Read the record length. + * Verify the record header. * * NB: Even though we use an XLogRecord pointer here, the whole record - * header might not fit on this page. xl_tot_len is the first field of the - * struct, so it must be on this page (the records are MAXALIGNed), but we - * cannot access any other fields until we've verified that we got the - * whole header. + * header might not fit on this page. */ record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ); - total_len = record->xl_tot_len; /* * If the whole record header is on this page, validate it immediately. @@ -664,19 +664,19 @@ restart: } else { - /* There may be no next page if it's too small. */ - if (total_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, total_len); + /* + * xl_tot_len is the first field of the struct, so it must be on this + * page (the records are MAXALIGNed), but we cannot access any other + * fields until we've verified that we got the whole header. + */ + if (!ValidXLogRecordLength(state, RecPtr, record)) goto err; - } - /* We'll validate the header once we have the next page. */ + gotheader = false; } + total_len = record->xl_tot_len; + /* * Try to find space to decode this record, if we can do so without * calling palloc. If we can't, we'll try again below after we've @@ -1119,25 +1119,80 @@ XLogReaderInvalReadState(XLogReaderState *state) state->readLen = 0; } +/* + * Validate record length of an XLOG record header. + * + * This is substantially a part of ValidXLogRecordHeader. But XLogReadRecord + * needs this separate from the function in case of a partial record header. + * + * Returns true if the xl_tot_len header field has a seemingly valid value, + * which means the caller can proceed reading to the following part of the + * record. + */ +static bool +ValidXLogRecordLength(XLogReaderState *state, XLogRecPtr RecPtr, + XLogRecord *record) +{ + if (record->xl_tot_len == 0) + { + char *p; + char *pe; + + /* + * We are almost sure reaching the end of WAL, make sure that the + * whole page after the record is filled with zeroes. + */ + p = (char *) record; + pe = p + XLOG_BLCKSZ - (RecPtr & (XLOG_BLCKSZ - 1)); + + while (p < pe && *p == 0) + p++; + + if (p == pe) + { + /* + * Consider it as end-of-WAL if all subsequent bytes of this page + * are zero. We don't bother checking the subsequent pages since + * they are not zeroed in the case of recycled segments. + */ + report_invalid_record(state, "empty record at %X/%X", + LSN_FORMAT_ARGS(RecPtr)); + + /* notify end-of-wal to callers */ + state->EndOfWAL = true; + return false; + } + } + + if (record->xl_tot_len < SizeOfXLogRecord) + { + report_invalid_record(state, + "invalid record length at %X/%X: expected at least %u, got %u", + LSN_FORMAT_ARGS(RecPtr), + (uint32) SizeOfXLogRecord, record->xl_tot_len); + return false; + } + + return true; +} + /* * Validate an XLOG record header. * - * This is just a convenience subroutine to avoid duplicated code in + * This is just a convenience subroutine to avoid duplicate code in * XLogReadRecord. It's not intended for use from anywhere else. + * + * Returns true if the header fields have the valid values and the caller can + * proceed reading to the following part of the record. */ static bool ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr, XLogRecPtr PrevRecPtr, XLogRecord *record, bool randAccess) { - if (record->xl_tot_len < SizeOfXLogRecord) - { - report_invalid_record(state, - "invalid record length at %X/%X: expected at least %u, got %u", - LSN_FORMAT_ARGS(RecPtr), - (uint32) SizeOfXLogRecord, record->xl_tot_len); + if (!ValidXLogRecordLength(state, RecPtr, record)) return false; - } + if (!RmgrIdIsValid(record->xl_rmid)) { report_invalid_record(state, @@ -1235,6 +1290,44 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, XLByteToSeg(recptr, segno, state->segcxt.ws_segsize); offset = XLogSegmentOffset(recptr, state->segcxt.ws_segsize); + StaticAssertStmt(XLOG_PAGE_MAGIC != 0, "XLOG_PAGE_MAGIC is zero"); + + if (hdr->xlp_magic == 0) + { + /* Regard an empty page as End-Of-WAL */ + int i; + + for (i = 0; i < XLOG_BLCKSZ && phdr[i] == 0; i++); + if (i == XLOG_BLCKSZ) + { + char fname[MAXFNAMELEN]; + + XLogFileName(fname, state->seg.ws_tli, segno, + state->segcxt.ws_segsize); + + /* + * Consider an empty page as end-of-WAL only when reading the first + * part of a record. + */ + if (state->currRecPtr / XLOG_BLCKSZ == recptr / XLOG_BLCKSZ) + { + report_invalid_record(state, + "empty page in WAL segment %s, offset %u", + fname, offset); + state->EndOfWAL = true; + } + else + report_invalid_record(state, + "empty page in WAL segment %s, offset %u while reading continuation record at %X/%X", + fname, offset, + LSN_FORMAT_ARGS(state->currRecPtr)); + + return false; + } + + /* The same condition will be caught as invalid magic number */ + } + if (hdr->xlp_magic != XLOG_PAGE_MAGIC) { char fname[MAXFNAMELEN]; @@ -1324,6 +1417,14 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr, fname, LSN_FORMAT_ARGS(recptr), offset); + + /* + * If the page address is less than expected we assume it is an unused + * page in a recycled segment. + */ + if (hdr->xlp_pageaddr < recptr) + state->EndOfWAL = true; + return false; } diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 1b48d7171a..3b56bd04a6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1678,7 +1678,7 @@ PerformWalRecovery(void) /* just have to read next record after CheckPoint */ Assert(xlogreader->ReadRecPtr == CheckPointLoc); replayTLI = CheckPointTLI; - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } if (record != NULL) @@ -1785,7 +1785,7 @@ PerformWalRecovery(void) } /* Else, try to fetch the next WAL record */ - record = ReadRecord(xlogprefetcher, LOG, false, replayTLI); + record = ReadRecord(xlogprefetcher, WARNING, false, replayTLI); } while (record != NULL); /* @@ -1839,12 +1839,19 @@ PerformWalRecovery(void) InRedo = false; } - else + else if (xlogreader->EndOfWAL) { /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); } + else + { + /* broken record found */ + ereport(WARNING, + errmsg("redo is skipped"), + errhint("This suggests WAL file corruption. You might need to check the database.")); + } /* * This check is intentionally after the above log messages that indicate @@ -3095,6 +3102,7 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, for (;;) { char *errormsg; + XLogRecPtr ErrRecPtr = InvalidXLogRecPtr; record = XLogPrefetcherReadRecord(xlogprefetcher, &errormsg); if (record == NULL) @@ -3116,6 +3124,18 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, { abortedRecPtr = xlogreader->abortedRecPtr; missingContrecPtr = xlogreader->missingContrecPtr; + ErrRecPtr = abortedRecPtr; + } + else + { + /* + * EndRecPtr is the LSN we tried to read but failed. In the + * case of decoding error, it is at the end of the failed + * record but we don't have a means for now to know EndRecPtr + * is pointing to which of the beginning or ending of the + * failed record. + */ + ErrRecPtr = xlogreader->EndRecPtr; } if (readFile >= 0) @@ -3126,13 +3146,15 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* * We only end up here without a message when XLogPageRead() - * failed - in that case we already logged something. In - * StandbyMode that only happens if we have been triggered, so we - * shouldn't loop anymore in that case. + * failed- in that case we already logged something. In StandbyMode + * that only happens if we have been triggered, so we shouldn't + * loop anymore in that case. When EndOfWAL is true, we don't emit + * the message immediately and instead will show it as a part of a + * decent end-of-wal message later. */ - if (errormsg) - ereport(emode_for_corrupt_record(emode, xlogreader->EndRecPtr), - (errmsg_internal("%s", errormsg) /* already translated */ )); + if (!xlogreader->EndOfWAL && errormsg) + ereport(emode_for_corrupt_record(emode, ErrRecPtr), + errmsg_internal("%s", errormsg) /* already translated */ ); } /* @@ -3163,11 +3185,14 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, /* Great, got a record */ return record; } - else + + Assert(ErrRecPtr != InvalidXLogRecPtr); + + /* No valid record available from this source */ + lastSourceFailed = true; + + if (!fetching_ckpt) { - /* No valid record available from this source */ - lastSourceFailed = true; - /* * If archive recovery was requested, but we were still doing * crash recovery, switch to archive recovery and retry using the @@ -3180,11 +3205,16 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, * we'd have no idea how far we'd have to replay to reach * consistency. So err on the safe side and give up. */ - if (!InArchiveRecovery && ArchiveRecoveryRequested && - !fetching_ckpt) + if (!InArchiveRecovery && ArchiveRecoveryRequested) { + /* + * We don't report this as LOG, since we don't stop recovery + * here + */ ereport(DEBUG1, - (errmsg_internal("reached end of WAL in pg_wal, entering archive recovery"))); + errmsg_internal("reached end of WAL at %X/%X on timeline %u in pg_wal during crash recovery, entering archive recovery", + LSN_FORMAT_ARGS(ErrRecPtr), + replayTLI)); InArchiveRecovery = true; if (StandbyModeRequested) EnableStandbyMode(); @@ -3205,12 +3235,24 @@ ReadRecord(XLogPrefetcher *xlogprefetcher, int emode, continue; } - /* In standby mode, loop back to retry. Otherwise, give up. */ - if (StandbyMode && !CheckForStandbyTrigger()) - continue; - else - return NULL; + /* + * recovery ended. + * + * Emit a decent message if we met end-of-WAL. Otherwise we should + * have already emitted an error message. + */ + if (xlogreader->EndOfWAL) + ereport(LOG, + errmsg("reached end of WAL at %X/%X on timeline %u", + LSN_FORMAT_ARGS(ErrRecPtr), replayTLI), + (errormsg ? errdetail_internal("%s", errormsg) : 0)); } + + /* In standby mode, loop back to retry. Otherwise, give up. */ + if (StandbyMode && !CheckForStandbyTrigger()) + continue; + else + return NULL; } } @@ -3307,11 +3349,17 @@ retry: case XLREAD_WOULDBLOCK: return XLREAD_WOULDBLOCK; case XLREAD_FAIL: + /* make sure we didn't exit standby mode without trigger */ + Assert(!StandbyMode || CheckForStandbyTrigger()); + if (readFile >= 0) close(readFile); readFile = -1; readLen = 0; readSource = XLOG_FROM_ANY; + + /* exit by promotion is not end-of-WAL */ + xlogreader->EndOfWAL = !StandbyMode; return XLREAD_FAIL; case XLREAD_SUCCESS: break; @@ -3979,7 +4027,11 @@ emode_for_corrupt_record(int emode, XLogRecPtr RecPtr) { static XLogRecPtr lastComplaint = 0; - if (readSource == XLOG_FROM_PG_WAL && emode == LOG) + /* + * readSource cannot be used in place of currentSource because readSource + * is reset on failure + */ + if (currentSource == XLOG_FROM_PG_WAL && emode <= WARNING) { if (RecPtr == lastComplaint) emode = DEBUG1; diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 728059518e..c69e33dbe7 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -497,10 +497,9 @@ WalReceiverMain(void) else if (len < 0) { ereport(LOG, - (errmsg("replication terminated by primary server"), - errdetail("End of WAL reached on timeline %u at %X/%X.", - startpointTLI, - LSN_FORMAT_ARGS(LogstreamResult.Write)))); + errmsg("replication terminated by primary server at %X/%X on timeline %u.", + LSN_FORMAT_ARGS(LogstreamResult.Write), + startpointTLI)); endofwal = true; break; } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 1f9403fc5c..1c45a25313 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -1309,9 +1309,16 @@ main(int argc, char **argv) exit(0); if (errormsg) - pg_fatal("error in WAL record at %X/%X: %s", - LSN_FORMAT_ARGS(xlogreader_state->ReadRecPtr), - errormsg); + { + if (xlogreader_state->EndOfWAL) + pg_log_info("end of WAL at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + else + pg_fatal("error in WAL record at %X/%X: %s", + LSN_FORMAT_ARGS(xlogreader_state->EndRecPtr), + errormsg); + } XLogReaderFree(xlogreader_state); diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl index 082cb8a589..b15e7a2ca3 100644 --- a/src/bin/pg_waldump/t/001_basic.pl +++ b/src/bin/pg_waldump/t/001_basic.pl @@ -147,10 +147,11 @@ command_fails_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile command_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile, $node->data_dir . '/pg_wal/' . $end_walfile ], qr/./, 'runs with start and end segment specified'); command_fails_like([ 'pg_waldump', '-p', $node->data_dir ], qr/error: no start WAL location given/, 'path option requires start location'); command_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn, '--end', $end_lsn ], qr/./, 'runs with path option and start and end locations'); -command_fails_like([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'falling off the end of the WAL results in an error'); + +command_checks_all([ 'pg_waldump', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/./], [qr/pg_waldump: end of WAL at.*: empty record at/], 'the end of the WAL doesn\'t result in an error'); command_like([ 'pg_waldump', '--quiet', $node->data_dir . '/pg_wal/' . $start_walfile ], qr/^$/, 'no output with --quiet option'); -command_fails_like([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], qr/error: error in WAL record at/, 'errors are shown with --quiet'); +command_checks_all([ 'pg_waldump', '--quiet', '-p', $node->data_dir, '--start', $start_lsn ], 0, [qr/^$/], [qr/pg_waldump: end of WAL at .*: empty record at .*/], 'end-status is shown with --quiet'); # Test for: Display a message that we're skipping data if `from` diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 2e9e5f43eb..7f879281f5 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -205,6 +205,7 @@ struct XLogReaderState */ XLogRecPtr ReadRecPtr; /* start of last record read */ XLogRecPtr EndRecPtr; /* end+1 of last record read */ + bool EndOfWAL; /* was the last attempt EOW? */ /* * Set at the end of recovery: the start point of a partial record at the diff --git a/src/test/recovery/t/035_recovery.pl b/src/test/recovery/t/035_recovery.pl new file mode 100644 index 0000000000..7107df6509 --- /dev/null +++ b/src/test/recovery/t/035_recovery.pl @@ -0,0 +1,130 @@ + +# Copyright (c) 2021-2023, PostgreSQL Global Development Group + +# Minimal test testing recovery process + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use IPC::Run; + +my $reached_eow_pat = "reached end of WAL at "; +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; + +my ($stdout, $stderr) = ('', ''); + +my $segsize = $node->safe_psql('postgres', + qq[SELECT setting FROM pg_settings WHERE name = 'wal_segment_size';]); + +# make sure no records afterwards go to the next segment +$node->safe_psql('postgres', qq[ + SELECT pg_switch_wal(); + CHECKPOINT; + CREATE TABLE t(); +]); +$node->stop('immediate'); + +# identify REDO WAL file +my $cmd = "pg_controldata -D " . $node->data_dir(); +$cmd = ['pg_controldata', '-D', $node->data_dir()]; +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stdout =~ /^Latest checkpoint's REDO WAL file:[ \t] *(.+)$/m, + "checkpoint file is identified"); +my $chkptfile = $1; + +# identify the last record +my $walfile = $node->data_dir() . "/pg_wal/$chkptfile"; +$cmd = ['pg_waldump', $walfile]; +$stdout = ''; +$stderr = ''; +my $lastrec; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +foreach my $l (split(/\r?\n/, $stdout)) +{ + $lastrec = $l; +} +ok(defined $lastrec, "last WAL record is extracted"); +ok($stderr =~ /end of WAL at ([0-9A-F\/]+): .* at \g1/, + "pg_waldump emits the correct ending message"); + +# read the last record LSN excluding leading zeroes +ok ($lastrec =~ /, lsn: 0\/0*([1-9A-F][0-9A-F]+),/, + "LSN of the last record identified"); +my $lastlsn = $1; + +# corrupt the last record +my $offset = hex($lastlsn) % $segsize; +open(my $segf, '+<', $walfile) or die "failed to open $walfile\n"; +seek($segf, $offset, 0); # zero xl_tot_len, leaving following bytes alone. +print $segf "\0\0\0\0"; +close($segf); + +# pg_waldump complains about the corrupted record +$stdout = ''; +$stderr = ''; +IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr; +ok($stderr =~ /error: error in WAL record at 0\/$lastlsn: .* at 0\/$lastlsn/, + "pg_waldump emits the correct error message"); + +# also server complains for the same reason +my $logstart = -s $node->logfile; +$node->start; +ok($node->wait_for_log( + "WARNING: invalid record length at 0/$lastlsn: expected at least 24, got 0", + $logstart), + "header error is correctly logged at $lastlsn"); + +# no end-of-wal message should be seen this time +ok(!find_in_log($node, $reached_eow_pat, $logstart), + "false log message is not emitted"); + +# Create streaming standby linking to primary +my $backup_name = 'my_backup'; +$node->backup($backup_name); +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node, $backup_name, has_streaming => 1); +$node_standby->start; +$node->safe_psql('postgres', 'CREATE TABLE t ()'); +my $primary_lsn = $node->lsn('write'); +$node->wait_for_catchup($node_standby, 'write', $primary_lsn); + +$node_standby->stop(); +$node->stop('immediate'); + +# crash restart the primary +$logstart = -s $node->logfile; +$node->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'primary properly emits end-of-WAL message'); + +# restart the standby +$logstart = -s $node_standby->logfile; +$node_standby->start(); +ok($node->wait_for_log($reached_eow_pat, $logstart), + 'standby properly emits end-of-WAL message'); + +$node_standby->stop(); +$node->stop(); + +done_testing(); + +#### helper routines +# find $pat in logfile of $node after $off-th byte +sub find_in_log +{ + my ($node, $pat, $off) = @_; + + $off = 0 unless defined $off; + my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile); + return 0 if (length($log) <= $off); + + $log = substr($log, $off); + + return $log =~ m/$pat/; +} diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl index f9acc83c7d..5a17f68512 100644 --- a/src/test/recovery/t/039_end_of_wal.pl +++ b/src/test/recovery/t/039_end_of_wal.pl @@ -258,16 +258,29 @@ my $prev_lsn; note "Single-page end-of-WAL detection"; ########################################################################### -# xl_tot_len is 0 (a common case, we hit trailing zeroes). -emit_message($node, 0); -$end_lsn = advance_out_of_record_splitting_zone($node); +# empty record without trailing garbage bytes until the page end - not error $node->stop('immediate'); my $log_size = -s $node->logfile; $node->start; +ok( $node->log_contains( + "LOG: reached end of WAL at.*\n.* DETAIL: empty record at", + $log_size), + "end-of-WAL by empty record"); + +# xl_tot_len is 0 with following garbage bytes in the page +emit_message($node, 0); +$end_lsn = advance_out_of_record_splitting_zone($node); +$node->stop('immediate'); +write_wal($node, $TLI, + # last byte in the page at $end_lsn + $end_lsn - ($end_lsn % $WAL_BLOCK_SIZE) + $WAL_BLOCK_SIZE - 1, + pack("c", 1)); # garbage byte +$log_size = -s $node->logfile; +$node->start; ok( $node->log_contains( "invalid record length at .*: expected at least 24, got 0", $log_size ), - "xl_tot_len zero"); + "zero xl_tot_len followed by garbage bytes"); # xl_tot_len is < 24 (presumably recycled garbage). emit_message($node, 0); @@ -328,7 +341,7 @@ note "Multi-page end-of-WAL detection, header is not split"; # This series of tests requires a valid xl_prev set in the record header # written to WAL. -# Good xl_prev, we hit zero page next (zero magic). +# Good xl_prev, we hit zero page next emit_message($node, 0); $prev_lsn = advance_out_of_record_splitting_zone($node); $end_lsn = emit_message($node, 0); @@ -337,8 +350,24 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "empty page"); + +# Good xl_prev, we hit zero page magic with following garbage bytes. +emit_message($node, 0); +$prev_lsn = advance_out_of_record_splitting_zone($node); +$end_lsn = emit_message($node, 0); +$node->stop('immediate'); +write_wal($node, $TLI, $end_lsn, + build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn)); +# place garbage at the end of the next page +write_wal($node, $TLI, + start_of_next_page(start_of_next_page($end_lsn)) - 1, + pack("i", 1)); +$log_size = -s $node->logfile; +$node->start; +ok( $node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), + "bad magic"); # Good xl_prev, we hit garbage page next (bad magic). emit_message($node, 0); @@ -442,8 +471,8 @@ write_wal($node, $TLI, $end_lsn, build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef)); $log_size = -s $node->logfile; $node->start; -ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size), - "xlp_magic zero (split record header)"); +ok( $node->log_contains("WARNING: empty page in WAL segment .*, offset .* while reading continuation record at .*", $log_size), + "zero page while reading a record (split record header)"); # And we'll also check xlp_pageaddr before any header checks. emit_message($node, 0); -- 2.39.3 ----Next_Part(Tue_Jan_23_13_13_56_2024_752)---- ^ permalink raw reply [nested|flat] 39+ messages in thread
* [PATCH] Introduce an option to make logical replication database specific. @ 2026-04-04 15:16 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 39+ messages in thread From: Antonin Houska @ 2026-04-04 15:16 UTC (permalink / raw) By default, the logical decoding assumes access to shared catalogs, so the snapshot builder needs to consider cluster-wide XIDs during startup. That in turn means that, if any transaction is already running (and has XID assigned), the snapshot builder needs to wait for its completion, as it does not know if that transaction performed catalog changes earlier. Possible problem with this concept is that if REPACK (CONCURRENTLY) is running in some database, backends running the same command in other databases get stuck until the first one has committed. Thus only as single backend in the cluster can run REPACK (CONCURRENTLY) at any time. Likewise, REPACK (CONCURRENTLY) can block walsenders starting on behalf of subscriptions throughout the cluster. This patch adds a new option to logical replication output plugin, to declare that it does not use shared catalogs (i.e. catalogs that can be changed by transactions running in other databases in the cluster). In that case, no snapshot the backend will use during the decoding needs to contain information about transactions running in other databases. Thus the snapshot builder only needs to wait for completion of transactions in the current database. Currently we only use this option in the REPACK background worker. It could possibly be used in the plugin for logical replication too, however that would need thorough analysis of that plugin. The patch bumps WAL version number, due to a new field in xl_running_xacts. --- contrib/pg_visibility/pg_visibility.c | 4 +- doc/src/sgml/logicaldecoding.sgml | 4 ++ src/backend/access/index/genam.c | 18 +++++++ src/backend/access/rmgrdesc/standbydesc.c | 2 + src/backend/access/transam/xlog.c | 2 +- src/backend/access/transam/xlogfuncs.c | 2 +- src/backend/postmaster/bgwriter.c | 2 +- src/backend/replication/logical/decode.c | 8 ++- src/backend/replication/logical/logical.c | 5 ++ src/backend/replication/logical/snapbuild.c | 52 ++++++++++++++++--- .../pgoutput_repack/pgoutput_repack.c | 7 +++ src/backend/replication/slot.c | 2 +- src/backend/storage/ipc/procarray.c | 23 +++++++- src/backend/storage/ipc/standby.c | 14 ++++- src/include/access/genam.h | 8 +++ src/include/access/xlog_internal.h | 2 +- src/include/replication/output_plugin.h | 1 + src/include/replication/snapbuild.h | 3 +- src/include/storage/procarray.h | 2 +- src/include/storage/standby.h | 3 +- src/include/storage/standbydefs.h | 1 + 21 files changed, 144 insertions(+), 21 deletions(-) diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c index dfab0b64cf5..d564bd2a00c 100644 --- a/contrib/pg_visibility/pg_visibility.c +++ b/contrib/pg_visibility/pg_visibility.c @@ -621,7 +621,7 @@ GetStrictOldestNonRemovableTransactionId(Relation rel) else if (rel == NULL || rel->rd_rel->relisshared) { /* Shared relation: take into account all running xids */ - runningTransactions = GetRunningTransactionData(); + runningTransactions = GetRunningTransactionData(InvalidOid); LWLockRelease(ProcArrayLock); LWLockRelease(XidGenLock); return runningTransactions->oldestRunningXid; @@ -632,7 +632,7 @@ GetStrictOldestNonRemovableTransactionId(Relation rel) * Normal relation: take into account xids running within the current * database */ - runningTransactions = GetRunningTransactionData(); + runningTransactions = GetRunningTransactionData(InvalidOid); LWLockRelease(ProcArrayLock); LWLockRelease(XidGenLock); return runningTransactions->oldestDatabaseRunningXid; diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 6dc49108997..28089d1053c 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -959,6 +959,7 @@ typedef struct OutputPluginOptions { OutputPluginOutputType output_type; bool receive_rewrites; + bool need_shared_catalogs; } OutputPluginOptions; </programlisting> <literal>output_type</literal> has to either be set to @@ -969,6 +970,9 @@ typedef struct OutputPluginOptions also be called for changes made by heap rewrites during certain DDL operations. These are of interest to plugins that handle DDL replication, but they require special handling. + <literal>need_shared_catalogs</literal> can be set to false if you are + sure the plugin functions do not access shared system catalogs. It can + speed up creation of replication slots that use this plugin. </para> <para> diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c index 1408989c568..df092dc999a 100644 --- a/src/backend/access/index/genam.c +++ b/src/backend/access/index/genam.c @@ -37,6 +37,14 @@ #include "utils/ruleutils.h" #include "utils/snapmgr.h" +/* + * If a backend is going to do logical decoding and if the output plugin does + * not need to access shared catalogs, setting this variable to false can make + * the decoding startup faster. In particular, the backend will not need to + * wait for completion of already running transactions in other databases. + */ +bool accessSharedCatalogsInDecoding = true; + /* ---------------------------------------------------------------- * general access method routines @@ -394,6 +402,16 @@ systable_beginscan(Relation heapRelation, SysScanDesc sysscan; Relation irel; + /* + * If this backend promised that it won't access shared catalogs during + * logical decoding, this seems to be the right place to check. + * + * XXX Should this be ereport(ERROR) ? + */ + Assert(!HistoricSnapshotActive() || + accessSharedCatalogsInDecoding || + !heapRelation->rd_rel->relisshared); + if (indexOK && !IgnoreSystemIndexes && !ReindexIsProcessingIndex(indexId)) diff --git a/src/backend/access/rmgrdesc/standbydesc.c b/src/backend/access/rmgrdesc/standbydesc.c index 0a291354ae2..685d1bdb024 100644 --- a/src/backend/access/rmgrdesc/standbydesc.c +++ b/src/backend/access/rmgrdesc/standbydesc.c @@ -41,6 +41,8 @@ standby_desc_running_xacts(StringInfo buf, xl_running_xacts *xlrec) for (i = 0; i < xlrec->subxcnt; i++) appendStringInfo(buf, " %u", xlrec->xids[xlrec->xcnt + i]); } + + appendStringInfo(buf, "; dbid: %u", xlrec->dbid); } void diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9e8999bbb61..1a6c1456d2c 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -7731,7 +7731,7 @@ CreateCheckPoint(int flags) * recovery we don't need to write running xact data. */ if (!shutdown && XLogStandbyInfoActive()) - LogStandbySnapshot(); + LogStandbySnapshot(InvalidOid); START_CRIT_SECTION(); diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 65bbaeda59c..0f5979691e6 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -245,7 +245,7 @@ pg_log_standby_snapshot(PG_FUNCTION_ARGS) (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("pg_log_standby_snapshot() can only be used if \"wal_level\" >= \"replica\""))); - recptr = LogStandbySnapshot(); + recptr = LogStandbySnapshot(InvalidOid); /* * As a convenience, return the WAL location of the last inserted record diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 1d8947774a9..a30de4262eb 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -289,7 +289,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) if (now >= timeout && last_snapshot_lsn <= GetLastImportantRecPtr()) { - last_snapshot_lsn = LogStandbySnapshot(); + last_snapshot_lsn = LogStandbySnapshot(InvalidOid); last_snapshot_ts = now; } } diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index 2be5d8d4bd9..1353d004aac 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -382,7 +382,13 @@ standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf) { xl_running_xacts *running = (xl_running_xacts *) XLogRecGetData(r); - SnapBuildProcessRunningXacts(builder, buf->origptr, running); + + /* + * If the plugin does not access shared catalogs, only keep + * track of transactions in the current database. + */ + SnapBuildProcessRunningXacts(builder, buf->origptr, running, + !ctx->options.need_shared_catalogs); /* * Abort all transactions that we keep track of, that are diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index f20a0fe70ad..89043a03497 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -289,6 +289,11 @@ StartupDecodingContext(List *output_plugin_options, ctx->write = do_write; ctx->update_progress = update_progress; + /* + * Assume true for safety, the startup callback can change that. + */ + ctx->options.need_shared_catalogs = true; + ctx->output_plugin_options = output_plugin_options; ctx->fast_forward = fast_forward; diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index b4269a3b102..054a7b16f43 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -125,6 +125,7 @@ #include <sys/stat.h> #include <unistd.h> +#include "access/genam.h" #include "access/heapam_xlog.h" #include "access/transam.h" #include "access/xact.h" @@ -170,7 +171,8 @@ static inline bool SnapBuildXidHasCatalogChanges(SnapBuild *builder, Transaction uint32 xinfo); /* xlog reading helper functions for SnapBuildProcessRunningXacts */ -static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running); +static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, + xl_running_xacts *running, bool db_specific); static void SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff); /* serialization functions */ @@ -1136,7 +1138,8 @@ SnapBuildXidHasCatalogChanges(SnapBuild *builder, TransactionId xid, * anymore. */ void -SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running) +SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running, + bool db_specific) { ReorderBufferTXN *txn; TransactionId xmin; @@ -1149,7 +1152,7 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact if (builder->state < SNAPBUILD_CONSISTENT) { /* returns false if there's no point in performing cleanup just yet */ - if (!SnapBuildFindSnapshot(builder, lsn, running)) + if (!SnapBuildFindSnapshot(builder, lsn, running, db_specific)) return; } else @@ -1166,8 +1169,13 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact * (see SnapBuildCommitTxn). Because of this, xmax can be lower than * xmin, which looks odd but is correct and actually more efficient, since * we hit fast paths in heapam_visibility.c. + * + * If database specific transaction info was used during startup, the info + * for the whole cluster can make xmin go backwards. That would be bad + * because we might no longer have older XIDs in ->committed. */ - builder->xmin = running->oldestRunningXid; + if (NormalTransactionIdFollows(running->oldestRunningXid, builder->xmin)) + builder->xmin = running->oldestRunningXid; /* Remove transactions we don't need to keep track off anymore */ SnapBuildPurgeOlderTxn(builder); @@ -1182,7 +1190,10 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact * the record we're reading now. */ xmin = ReorderBufferGetOldestXmin(builder->reorder); - if (xmin == InvalidTransactionId) + /* + * Like above, do not let slot xmin go backwards. + */ + if (xmin == InvalidTransactionId && !db_specific) xmin = running->oldestRunningXid; elog(DEBUG3, "xmin: %u, xmax: %u, oldest running: %u, oldest xmin: %u", builder->xmin, builder->xmax, running->oldestRunningXid, xmin); @@ -1238,7 +1249,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact * using the xl_running_xacts record. */ static bool -SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running) +SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running, + bool db_specific) { /* --- * Build catalog decoding snapshot incrementally using information about @@ -1265,6 +1277,28 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn * --- */ + if (db_specific) + { + /* + * If we should only keep track of transactions running in the current + * database, we need transaction info from exactly that + * database. Invalid 'dbid' (all databases) would satisfy our needs + * too, but it could make us wait unnecessarily for completion of + * transactions running in other databases. + */ + if (running->dbid != MyDatabaseId) + { + LogStandbySnapshot(MyDatabaseId); + return false; + } + + /* + * We'd better be able to check during scan if the plugin does not + * lie. + */ + accessSharedCatalogsInDecoding = false; + } + /* * xl_running_xacts record is older than what we can use, we might not * have all necessary catalog rows anymore. @@ -1465,7 +1499,11 @@ SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff) */ if (!RecoveryInProgress()) { - LogStandbySnapshot(); + /* + * If the last transaction info was about specific database, so needs + * to be the next one - at least until we're in the CONSISTENT state. + */ + LogStandbySnapshot(running->dbid); } } diff --git a/src/backend/replication/pgoutput_repack/pgoutput_repack.c b/src/backend/replication/pgoutput_repack/pgoutput_repack.c index 221260ca0a8..bb3a880a4c3 100644 --- a/src/backend/replication/pgoutput_repack/pgoutput_repack.c +++ b/src/backend/replication/pgoutput_repack/pgoutput_repack.c @@ -52,6 +52,13 @@ repack_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt, /* Probably unnecessary, as we don't use the SQL interface ... */ opt->output_type = OUTPUT_PLUGIN_BINARY_OUTPUT; + /* + * By declaring that our output plugin does not need shared catalogs, we + * avoid waiting for completion of transactions running in other databases + * than the one we're connected to. + */ + opt->need_shared_catalogs = false; + if (ctx->output_plugin_options != NIL) { ereport(ERROR, diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index a9092fc2382..9533515a63b 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -1762,7 +1762,7 @@ ReplicationSlotReserveWal(void) XLogRecPtr flushptr; /* make sure we have enough information to start */ - flushptr = LogStandbySnapshot(); + flushptr = LogStandbySnapshot(InvalidOid); /* and make sure it's fsynced to disk */ XLogFlush(flushptr); diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index cc207cb56e3..92ed34128e5 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -2631,9 +2631,11 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc) * * Note that if any transaction has overflowed its cached subtransactions * then there is no real need include any subtransactions. + * + * If 'dbid' is valid, only gather transactions running in that database. */ RunningTransactions -GetRunningTransactionData(void) +GetRunningTransactionData(Oid dbid) { /* result workspace */ static RunningTransactionsData CurrentRunningXactsData; @@ -2708,6 +2710,18 @@ GetRunningTransactionData(void) if (!TransactionIdIsValid(xid)) continue; + /* + * Filter by database OID if requested. + */ + if (OidIsValid(dbid)) + { + int pgprocno = arrayP->pgprocnos[index]; + PGPROC *proc = &allProcs[pgprocno]; + + if (proc->databaseId != dbid) + continue; + } + /* * Be careful not to exclude any xids before calculating the values of * oldestRunningXid and suboverflowed, since these are used to clean @@ -2758,6 +2772,12 @@ GetRunningTransactionData(void) PGPROC *proc = &allProcs[pgprocno]; int nsubxids; + /* + * Filter by database OID if requested. + */ + if (OidIsValid(dbid) && proc->databaseId != dbid) + continue; + /* * Save subtransaction XIDs. Other backends can't add or remove * entries while we're holding XidGenLock. @@ -2791,6 +2811,7 @@ GetRunningTransactionData(void) * increases if slots do. */ + CurrentRunningXacts->dbid = dbid; CurrentRunningXacts->xcnt = count - subcount; CurrentRunningXacts->subxcnt = subcount; CurrentRunningXacts->subxid_status = suboverflowed ? SUBXIDS_IN_SUBTRANS : SUBXIDS_IN_ARRAY; diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index de9092fdf5b..c653ea742bc 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -1188,6 +1188,14 @@ standby_redo(XLogReaderState *record) xl_running_xacts *xlrec = (xl_running_xacts *) XLogRecGetData(record); RunningTransactionsData running; + /* + * Records issued for specific database are not suitable for physical + * replication because that affects the whole cluster. In particular, + * the list of XID is probably incomplete here. + */ + if (OidIsValid(xlrec->dbid)) + return; + running.xcnt = xlrec->xcnt; running.subxcnt = xlrec->subxcnt; running.subxid_status = xlrec->subxid_overflow ? SUBXIDS_MISSING : SUBXIDS_IN_ARRAY; @@ -1277,11 +1285,12 @@ standby_redo(XLogReaderState *record) * as there's no independent knob to just enable logical decoding. For * details of how this is used, check snapbuild.c's introductory comment. * + * If 'dbid' is valid, only gather transactions running in that database. * * Returns the RecPtr of the last inserted record. */ XLogRecPtr -LogStandbySnapshot(void) +LogStandbySnapshot(Oid dbid) { XLogRecPtr recptr; RunningTransactions running; @@ -1314,7 +1323,7 @@ LogStandbySnapshot(void) * Log details of all in-progress transactions. This should be the last * record we write, because standby will open up when it sees this. */ - running = GetRunningTransactionData(); + running = GetRunningTransactionData(dbid); /* * GetRunningTransactionData() acquired ProcArrayLock, we must release it. @@ -1358,6 +1367,7 @@ LogCurrentRunningXacts(RunningTransactions CurrRunningXacts) xl_running_xacts xlrec; XLogRecPtr recptr; + xlrec.dbid = CurrRunningXacts->dbid; xlrec.xcnt = CurrRunningXacts->xcnt; xlrec.subxcnt = CurrRunningXacts->subxcnt; xlrec.subxid_overflow = (CurrRunningXacts->subxid_status != SUBXIDS_IN_ARRAY); diff --git a/src/include/access/genam.h b/src/include/access/genam.h index b69320a7fc8..56c573399ab 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -136,6 +136,14 @@ typedef struct IndexOrderByDistance bool isnull; } IndexOrderByDistance; +/* + * Is the backend interested in shared catalogs when performing logical + * decoding? + * + * XXX Is there a better place for this declaration? + */ +extern bool accessSharedCatalogsInDecoding; + /* * generalized index_ interface routines (in indexam.c) */ diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h index 10c18d39ff8..13ae3ad4fbb 100644 --- a/src/include/access/xlog_internal.h +++ b/src/include/access/xlog_internal.h @@ -32,7 +32,7 @@ /* * Each page of XLOG file has a header like this: */ -#define XLOG_PAGE_MAGIC 0xD11E /* can be used as WAL version indicator */ +#define XLOG_PAGE_MAGIC 0xD11F /* can be used as WAL version indicator */ typedef struct XLogPageHeaderData { diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h index 842fcde67f9..917f3cff232 100644 --- a/src/include/replication/output_plugin.h +++ b/src/include/replication/output_plugin.h @@ -27,6 +27,7 @@ typedef struct OutputPluginOptions { OutputPluginOutputType output_type; bool receive_rewrites; + bool need_shared_catalogs; } OutputPluginOptions; /* diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h index ccded021433..82552a97997 100644 --- a/src/include/replication/snapbuild.h +++ b/src/include/replication/snapbuild.h @@ -93,7 +93,8 @@ extern void SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid, XLogRecPtr lsn, struct xl_heap_new_cid *xlrec); extern void SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, - struct xl_running_xacts *running); + struct xl_running_xacts *running, + bool db_specific); extern void SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn); extern bool SnapBuildSnapshotExists(XLogRecPtr lsn); diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h index abdf021e66e..377b3060b9f 100644 --- a/src/include/storage/procarray.h +++ b/src/include/storage/procarray.h @@ -49,7 +49,7 @@ extern bool ProcArrayInstallImportedXmin(TransactionId xmin, VirtualTransactionId *sourcevxid); extern bool ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc); -extern RunningTransactions GetRunningTransactionData(void); +extern RunningTransactions GetRunningTransactionData(Oid dbid); extern bool TransactionIdIsInProgress(TransactionId xid); extern TransactionId GetOldestNonRemovableTransactionId(Relation rel); diff --git a/src/include/storage/standby.h b/src/include/storage/standby.h index 6a314c693cd..8715c08e94f 100644 --- a/src/include/storage/standby.h +++ b/src/include/storage/standby.h @@ -126,6 +126,7 @@ typedef enum typedef struct RunningTransactionsData { + Oid dbid; /* only track xacts in this database */ int xcnt; /* # of xact ids in xids[] */ int subxcnt; /* # of subxact ids in xids[] */ subxids_array_status subxid_status; @@ -143,7 +144,7 @@ typedef RunningTransactionsData *RunningTransactions; extern void LogAccessExclusiveLock(Oid dbOid, Oid relOid); extern void LogAccessExclusiveLockPrepare(void); -extern XLogRecPtr LogStandbySnapshot(void); +extern XLogRecPtr LogStandbySnapshot(Oid dbid); extern void LogStandbyInvalidations(int nmsgs, SharedInvalidationMessage *msgs, bool relcacheInitFileInval); diff --git a/src/include/storage/standbydefs.h b/src/include/storage/standbydefs.h index 231d251fd51..e75b7078766 100644 --- a/src/include/storage/standbydefs.h +++ b/src/include/storage/standbydefs.h @@ -46,6 +46,7 @@ typedef struct xl_standby_locks */ typedef struct xl_running_xacts { + Oid dbid; /* only track xacts in this database */ int xcnt; /* # of xact ids in xids[] */ int subxcnt; /* # of subxact ids in xids[] */ bool subxid_overflow; /* snapshot overflowed, subxids missing */ -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 39+ messages in thread
end of thread, other threads:[~2026-04-04 15:16 UTC | newest] Thread overview: 39+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-02-28 06:52 [PATCH v15] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v7] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v14] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v2] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v12] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v9] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v10] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v11] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v16] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v13] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v5] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v17] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v11] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v8] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v6] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v3] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2020-02-28 06:52 [PATCH v18] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-07-07 02:51 [PATCH v19] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-07-07 02:51 [PATCH] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-07-07 02:51 [PATCH] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-07-07 02:51 [PATCH v21] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-15 04:41 [PATCH v22] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-15 04:41 [PATCH v22] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-15 04:41 [PATCH] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-15 04:41 [PATCH] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-30 02:51 [PATCH v23] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-30 02:51 [PATCH v23] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2022-11-30 02:51 [PATCH v24] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v28] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v28] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v25] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v27] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v29] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v27] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v28] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v29] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2023-03-07 05:55 [PATCH v29] Make End-Of-Recovery error less scary Kyotaro Horiguchi <horikyota.ntt@gmail.com> 2026-04-04 15:16 [PATCH] Introduce an option to make logical replication database specific. Antonin Houska <ah@cybertec.at>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox