pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17928: Standby fails to decode WAL on termination of primary
86+ messages / 8 participants
[nested] [flat]
* BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-05-11 08:00 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 86+ messages in thread
From: PG Bug reporting form @ 2023-05-11 08:00 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: exclusion@gmail.com
The following bug has been logged on the website:
Bug reference: 17928
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 15.2
Operating system: Ubuntu 22.04
Description:
When a primary server is stopped, standby might fail to replay WAL with
the memory allocation error, for example:
2023-05-11 08:42:38.254 MSK [1706654] LOG: recovery restart point at
7/12012E88
2023-05-11 08:42:38.254 MSK [1706654] DETAIL: Last completed transaction
was at log time 2023-05-11 08:42:38.245399+03.
2023-05-11 08:42:38.420 MSK [1706654] LOG: restartpoint starting: wal
2023-05-11 08:42:38.451 MSK [1725929] FATAL: could not receive data from
WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2023-05-11 08:42:38.468 MSK [1706656] FATAL: invalid memory alloc request
size 2021163525
2023-05-11 08:42:38.469 MSK [1706643] LOG: startup process (PID 1706656)
exited with exit code 1
2023-05-11 08:42:38.469 MSK [1706643] LOG: terminating any other active
server processes
2023-05-11 08:42:38.469 MSK [1706643] LOG: shutting down due to startup
process failure
2023-05-11 08:42:38.470 MSK [1706643] LOG: database system is shut down
The call stack of the error:
...
#6 0x000055cfabab8e97 in palloc_extended (size=2021163525, flags=2) at
mcxt.c:1143
#7 0x000055cfab456ae0 in XLogReadRecordAlloc (state=0x55cfacef4a08,
xl_tot_len=2021161080,
allow_oversized=true) at xlogreader.c:524
#8 0x000055cfab456dee in XLogDecodeNextRecord (state=0x55cfacef4a08,
nonblocking=false) at xlogreader.c:689
#9 0x000055cfab4575c0 in XLogReadAhead (state=0x55cfacef4a08,
nonblocking=false) at xlogreader.c:960
#10 0x000055cfab45519d in XLogPrefetcherNextBlock
(pgsr_private=94350447959632, lsn=0x55cfacf4bc58)
at xlogprefetcher.c:496
#11 0x000055cfab454b1c in lrq_prefetch (lrq=0x55cfacf4bb08) at
xlogprefetcher.c:256
#12 0x000055cfab454cdf in lrq_complete_lsn (lrq=0x55cfacf4bb08,
lsn=2976923632) at xlogprefetcher.c:294
#13 0x000055cfab455df2 in XLogPrefetcherReadRecord
(prefetcher=0x55cfacef5e50, errmsg=0x7ffc2e410458)
at xlogprefetcher.c:1039
#14 0x000055cfab45f136 in ReadRecord (xlogprefetcher=0x55cfacef5e50,
emode=15, fetching_ckpt=false,
replayTLI=1) at xlogrecovery.c:3047
#15 0x000055cfab45c922 in PerformWalRecovery () at xlogrecovery.c:1754
#16 0x000055cfab448869 in StartupXLOG () at xlog.c:5300
#17 0x000055cfab7d151a in StartupProcessMain () at startup.c:267
#18 0x000055cfab7c3f80 in AuxiliaryProcessMain (auxtype=StartupProcess) at
auxprocess.c:141
#19 0x000055cfab7cfb4b in StartChildProcess (type=StartupProcess) at
postmaster.c:5429
#20 0x000055cfab7ca5c0 in PostmasterMain (argc=4, argv=0x55cfacef3e40) at
postmaster.c:1470
#21 0x000055cfab6be302 in main (argc=4, argv=0x55cfacef3e40) at main.c:202
(gdb) frame 8
#8 0x000055cfab456dee in XLogDecodeNextRecord (state=0x55cfacef4a08,
nonblocking=false) at xlogreader.c:689
689 decoded = XLogReadRecordAlloc(state,
(gdb) print/x *record
$5 = {xl_tot_len = 0x78787878, xl_xid = 0x78787878, xl_prev =
0x7878787878787878, xl_info = 0x0,
xl_rmid = 0x10, xl_crc = 0x0}
(gdb) print/x state->NextRecPtr
$6 = 0xb1703ff0
hexdump -C .../standby_data/pgdata/pg_wal/0000000100000000000000B1
00703fe0 00 00 04 40 00 00 0c 56 00 00 00 00 00 00 00 00 ...@...V........
00703ff0 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 xxxxxxxxxxxxxxxx
00704000 10 d1 05 00 01 00 00 00 00 40 70 30 00 00 00 00 .........@p0....
00704010 db 00 00 00 00 00 00 00 78 78 78 78 78 78 78 78 ........xxxxxxxx
00704020 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 xxxxxxxxxxxxxxxx
*
007040f0 09 00 00 00 00 00 00 00 33 02 00 00 1b 0d 00 00 ........3.......
hexdump -C .../primary_data/pgdata/pg_wal/0000000100000000000000B1
00703fe0 00 00 04 40 00 00 0c 56 00 00 00 00 00 00 00 00 ...@...V........
00703ff0 33 02 00 00 54 2a 00 00 e0 3e 70 b1 00 00 00 00 3...T*...>p.....
00704000 10 d1 05 00 01 00 00 00 00 40 70 b1 00 00 00 00 .........@p.....
00704010 23 02 00 00 00 00 00 00 80 0a 00 00 55 0f 07 2c #...........U..,
00704020 00 60 02 02 7f 06 00 00 05 00 00 00 f7 85 01 00 .`..............
00704030 00 00 00 00 ff 03 02 00 02 08 18 00 01 00 00 00 ................
00704040 e0 07 00 00 78 78 78 78 78 78 78 78 78 78 78 78 ....xxxxxxxxxxxx
00704050 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 78 xxxxxxxxxxxxxxxx
*
00704230 78 78 78 78 78 78 78 78 01 00 00 00 00 00 00 00 xxxxxxxx........
(In this case, rows with 500 'x' were inserted when the primary was
stopped.)
`git bisect` for this behavior blames 3f1ce9734 (where
XLogDecodeNextRecord() -> XLogReadRecordAlloc() call was introduced).
A reproducer for the anomaly to follow.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-05-11 18:00 Alexander Lakhin <exclusion@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 86+ messages in thread
From: Alexander Lakhin @ 2023-05-11 18:00 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
11.05.2023 11:00, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 17928
> ...
> `git bisect` for this behavior blames 3f1ce9734 (where
> XLogDecodeNextRecord() -> XLogReadRecordAlloc() call was introduced).
>
> A reproducer for the anomaly to follow.
The TAP test that demonstrates the issue is attached. To catch the failure
faster, I place it in multiple directories src/test/recoveryX/t, add
minimal Makefiles, and run (on tmpfs):
for ((i=1;i<=10;i++)); do echo "iteration $i"; NO_TEMP_INSTALL=1 parallel --halt now,fail=1 -j7 --linebuffer --tag make
-s check -C src/test/{} ::: recovery1 recovery2 recovery3 recovery4 recovery5 recovery6 recovery7 || break; done
iteration 1
recovery1 +++ tap check in src/test/recovery1 +++
recovery2 +++ tap check in src/test/recovery2 +++
recovery3 +++ tap check in src/test/recovery3 +++
recovery4 +++ tap check in src/test/recovery4 +++
recovery5 +++ tap check in src/test/recovery5 +++
recovery6 +++ tap check in src/test/recovery6 +++
recovery7 +++ tap check in src/test/recovery7 +++
...
recovery5 # Restarting primary instance (49)
recovery3 # Restarting primary instance (49)
recovery7 # Restarting primary instance (49)
recovery2 Bailout called. Further testing stopped: pg_ctl stop failed
recovery2 FAILED--Further testing stopped: pg_ctl stop failed
recovery2 make: *** [Makefile:6: check] Error 255
parallel: This job failed:
make -s check -C src/test/recovery2
tail src/test/recovery2/tmp_check/log/099_restart_with_stanby_standby.log
2023-05-11 20:19:22.247 MSK [2046385] DETAIL: End of WAL reached on timeline 1 at 3/64BDFF8.
2023-05-11 20:19:22.247 MSK [2046385] FATAL: could not send end-of-streaming message to primary: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
no COPY in progress
2023-05-11 20:19:22.248 MSK [2037134] FATAL: invalid memory alloc request size 2021163525
2023-05-11 20:19:22.248 MSK [2037114] LOG: startup process (PID 2037134) exited with exit code 1
2023-05-11 20:19:22.248 MSK [2037114] LOG: terminating any other active server processes
2023-05-11 20:19:22.248 MSK [2037114] LOG: shutting down due to startup process failure
2023-05-11 20:19:22.249 MSK [2037114] LOG: database system is shut down
Best regards,
Alexander
Attachments:
[application/x-perl] 099_restart_with_stanby.pl (1.7K, ../../72bd036d-4f2a-8d50-b56e-6b1e3b9ba0a9@gmail.com/2-099_restart_with_stanby.pl)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-05-15 03:38 Thomas Munro <thomas.munro@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-05-15 03:38 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
On Fri, May 12, 2023 at 6:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> 2023-05-11 20:19:22.248 MSK [2037134] FATAL: invalid memory alloc request size 2021163525
> 2023-05-11 20:19:22.248 MSK [2037114] LOG: startup process (PID 2037134) exited with exit code 1
Thanks Alexander. Looking into this. I think it is probably
something like: recycled standby pages are not zeroed (something we
already needed to do something about[1]), and when we read a recycled
garbage size (like your "xxxx") at the end of a page at an offset
where we don't have a full record header on one page, we skip the
ValidXLogRecordHeader() call (and always did), but the check in
allocate_recordbuf() which previously handled that "gracefully" (well,
it would try to allocate up to 1GB bogusly, but it wouldn't try to
allocate more than that and ereport) is a bit too late. I probably
need to add an earlier not-too-big validation. Thinking.
[1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-07-10 20:00 Noah Misch <noah@leadboat.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Noah Misch @ 2023-07-10 20:00 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-bugs@lists.postgresql.org
On Mon, May 15, 2023 at 03:38:17PM +1200, Thomas Munro wrote:
> On Fri, May 12, 2023 at 6:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> > 2023-05-11 20:19:22.248 MSK [2037134] FATAL: invalid memory alloc request size 2021163525
> > 2023-05-11 20:19:22.248 MSK [2037114] LOG: startup process (PID 2037134) exited with exit code 1
>
> Thanks Alexander. Looking into this. I think it is probably
> something like: recycled standby pages are not zeroed (something we
> already needed to do something about[1]), and when we read a recycled
> garbage size (like your "xxxx") at the end of a page at an offset
> where we don't have a full record header on one page, we skip the
> ValidXLogRecordHeader() call (and always did), but the check in
> allocate_recordbuf() which previously handled that "gracefully" (well,
> it would try to allocate up to 1GB bogusly, but it wouldn't try to
> allocate more than that and ereport) is a bit too late. I probably
> need to add an earlier not-too-big validation. Thinking.
I agree about an earlier not-too-big validation. Like the attached? I
haven't tested it with Alexander's recipe or pondered it thoroughly.
> [1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
Regarding [1], is it still worth zeroing recycled pages on standbys and/or
reading the whole header before allocating xl_tot_len? (Are there benefits
other than avoiding a 1G backend allocation or 4G frontend allocation, or is
that benefit worth the cycles?)
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Validate xl_tot_len early enough. FIXME
This also allows distinguishing later OOM from an invalid record.
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e..ec754db 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -201,25 +201,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -641,7 +622,7 @@ restart:
Assert(pageHeaderSize <= readOff);
/*
- * Read the record length.
+ * Validate the record length.
*
* NB: Even though we use an XLogRecord pointer here, the whole record
* header might not fit on this page. xl_tot_len is the first field of the
@@ -651,35 +632,42 @@ restart:
*/
record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
total_len = record->xl_tot_len;
+ if (total_len < SizeOfXLogRecord)
+ {
+ report_invalid_record(state,
+ "invalid record length at %X/%X: expected at least %u, got %u",
+ LSN_FORMAT_ARGS(RecPtr),
+ (uint32) SizeOfXLogRecord, total_len);
+ goto err;
+ }
+#ifndef FRONTEND
/*
- * If the whole record header is on this page, validate it immediately.
- * Otherwise do just a basic sanity check on xl_tot_len, and validate the
- * rest of the header after reading it from the next page. The xl_tot_len
- * check is necessary here to ensure that we enter the "Need to reassemble
- * record" code path below; otherwise we might fail to apply
- * ValidXLogRecordHeader at all.
+ * We may be looking at a random uint32 read from a recycled segment. For
+ * a standby, a hard allocation failure would cause the instance to stop
+ * suddenly, preventing it to retry fetching WAL from one of its sources
+ * which could allow it to move on with replay without a manual restart.
+ * If the data comes from a past recycled segment and is still valid, then
+ * the allocation may succeed but record checks are going to fail so this
+ * would be short-lived.
*/
- if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord)
- {
- if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record,
- randAccess))
- goto err;
- gotheader = true;
- }
- else
+ if (!AllocSizeIsValid(DecodeXLogRecordRequiredSpace(total_len)))
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
- gotheader = false;
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
}
+#endif
+
+ /*
+ * If the whole record header is on this page, validate it immediately.
+ * Otherwise, validate it after reading it from the next page.
+ */
+ gotheader = targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord ? true : false;
+ if (gotheader &&
+ !ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record,
+ randAccess))
+ goto err;
/*
* Find space to decode this record. Don't allow oversized allocation if
@@ -723,9 +711,9 @@ restart:
if (total_len > state->readRecordBufSize &&
!allocate_recordbuf(state, total_len))
{
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u",
+ total_len);
goto err;
}
@@ -1113,14 +1101,6 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
XLogRecPtr PrevRecPtr, XLogRecord *record,
bool randAccess)
{
- if (record->xl_tot_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, record->xl_tot_len);
- return false;
- }
if (!RmgrIdIsValid(record->xl_rmid))
{
report_invalid_record(state,
@@ -1169,10 +1149,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
* record (other than to the minimal extent of computing the amount of
* data to read in) until we've checked the CRCs.
*
- * We assume all of the record (that is, xl_tot_len bytes) has been read
- * into memory at *record. Also, ValidXLogRecordHeader() has accepted the
- * record's header, which means in particular that xl_tot_len is at least
- * SizeOfXLogRecord.
+ * We assume all of the record (that is, xl_tot_len bytes) has been read into
+ * memory at *record. Also, xl_tot_len is at least SizeOfXLogRecord, and
+ * ValidXLogRecordHeader() has passed.
*/
static bool
ValidXLogRecord(XLogReaderState *state, XLogRecord *record, XLogRecPtr recptr)
Attachments:
[text/plain] xl_tot_len-validate-v1.patch (5.9K, ../../20230710200012.GB3476234@rfd.leadboat.com/2-xl_tot_len-validate-v1.patch)
download | inline diff:
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Validate xl_tot_len early enough. FIXME
This also allows distinguishing later OOM from an invalid record.
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e..ec754db 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -201,25 +201,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -641,7 +622,7 @@ restart:
Assert(pageHeaderSize <= readOff);
/*
- * Read the record length.
+ * Validate the record length.
*
* NB: Even though we use an XLogRecord pointer here, the whole record
* header might not fit on this page. xl_tot_len is the first field of the
@@ -651,35 +632,42 @@ restart:
*/
record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
total_len = record->xl_tot_len;
+ if (total_len < SizeOfXLogRecord)
+ {
+ report_invalid_record(state,
+ "invalid record length at %X/%X: expected at least %u, got %u",
+ LSN_FORMAT_ARGS(RecPtr),
+ (uint32) SizeOfXLogRecord, total_len);
+ goto err;
+ }
+#ifndef FRONTEND
/*
- * If the whole record header is on this page, validate it immediately.
- * Otherwise do just a basic sanity check on xl_tot_len, and validate the
- * rest of the header after reading it from the next page. The xl_tot_len
- * check is necessary here to ensure that we enter the "Need to reassemble
- * record" code path below; otherwise we might fail to apply
- * ValidXLogRecordHeader at all.
+ * We may be looking at a random uint32 read from a recycled segment. For
+ * a standby, a hard allocation failure would cause the instance to stop
+ * suddenly, preventing it to retry fetching WAL from one of its sources
+ * which could allow it to move on with replay without a manual restart.
+ * If the data comes from a past recycled segment and is still valid, then
+ * the allocation may succeed but record checks are going to fail so this
+ * would be short-lived.
*/
- if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord)
- {
- if (!ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record,
- randAccess))
- goto err;
- gotheader = true;
- }
- else
+ if (!AllocSizeIsValid(DecodeXLogRecordRequiredSpace(total_len)))
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
- gotheader = false;
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
}
+#endif
+
+ /*
+ * If the whole record header is on this page, validate it immediately.
+ * Otherwise, validate it after reading it from the next page.
+ */
+ gotheader = targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord ? true : false;
+ if (gotheader &&
+ !ValidXLogRecordHeader(state, RecPtr, state->DecodeRecPtr, record,
+ randAccess))
+ goto err;
/*
* Find space to decode this record. Don't allow oversized allocation if
@@ -723,9 +711,9 @@ restart:
if (total_len > state->readRecordBufSize &&
!allocate_recordbuf(state, total_len))
{
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u",
+ total_len);
goto err;
}
@@ -1113,14 +1101,6 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
XLogRecPtr PrevRecPtr, XLogRecord *record,
bool randAccess)
{
- if (record->xl_tot_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, record->xl_tot_len);
- return false;
- }
if (!RmgrIdIsValid(record->xl_rmid))
{
report_invalid_record(state,
@@ -1169,10 +1149,9 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
* record (other than to the minimal extent of computing the amount of
* data to read in) until we've checked the CRCs.
*
- * We assume all of the record (that is, xl_tot_len bytes) has been read
- * into memory at *record. Also, ValidXLogRecordHeader() has accepted the
- * record's header, which means in particular that xl_tot_len is at least
- * SizeOfXLogRecord.
+ * We assume all of the record (that is, xl_tot_len bytes) has been read into
+ * memory at *record. Also, xl_tot_len is at least SizeOfXLogRecord, and
+ * ValidXLogRecordHeader() has passed.
*/
static bool
ValidXLogRecord(XLogReaderState *state, XLogRecord *record, XLogRecPtr recptr)
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-07-11 06:54 Kyotaro Horiguchi <horikyota.ntt@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 86+ messages in thread
From: Kyotaro Horiguchi @ 2023-07-11 06:54 UTC (permalink / raw)
To: noah@leadboat.com; +Cc: thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
At Mon, 10 Jul 2023 13:00:12 -0700, Noah Misch <noah@leadboat.com> wrote in
> On Mon, May 15, 2023 at 03:38:17PM +1200, Thomas Munro wrote:
> > On Fri, May 12, 2023 at 6:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> > > 2023-05-11 20:19:22.248 MSK [2037134] FATAL: invalid memory alloc request size 2021163525
> > > 2023-05-11 20:19:22.248 MSK [2037114] LOG: startup process (PID 2037134) exited with exit code 1
> >
> > Thanks Alexander. Looking into this. I think it is probably
> > something like: recycled standby pages are not zeroed (something we
> > already needed to do something about[1]), and when we read a recycled
> > garbage size (like your "xxxx") at the end of a page at an offset
> > where we don't have a full record header on one page, we skip the
> > ValidXLogRecordHeader() call (and always did), but the check in
> > allocate_recordbuf() which previously handled that "gracefully" (well,
> > it would try to allocate up to 1GB bogusly, but it wouldn't try to
> > allocate more than that and ereport) is a bit too late. I probably
> > need to add an earlier not-too-big validation. Thinking.
>
> I agree about an earlier not-too-big validation. Like the attached? I
> haven't tested it with Alexander's recipe or pondered it thoroughly.
I like the patch for its code clean up, but I'm afraid that it removes
the existing record length check when reading continued
pages. However, I'm unsure about merely adding a check for too-long
records, due to the potential for requesting an excessively large
amount of memory, even if it will be released shortly.
> > [1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
>
> Regarding [1], is it still worth zeroing recycled pages on standbys and/or
> reading the whole header before allocating xl_tot_len? (Are there benefits
> other than avoiding a 1G backend allocation or 4G frontend allocation, or is
> that benefit worth the cycles?)
I believe reading the whole header is the most sensible approach as it
can prevent unnecessary memory requests. Another potential solution
(or hack) for this specific case is to let XLogWalRcvFlush write a
finalizing ((uint32)0) when dying is true. This stabilizes the
behavior to "invalid record length.. got 0" when running the TAP test.
regards.
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
[text/x-patch] write_finalize_tot_len.diff (570B, ../../20230711.155414.939136339819647605.horikyota.ntt@gmail.com/2-write_finalize_tot_len.diff)
download | inline diff:
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index feff709435..a570d56ada 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -989,6 +989,15 @@ XLogWalRcvFlush(bool dying, TimeLineID tli)
{
Assert(tli != 0);
+ if (dying)
+ {
+ int buflen = sizeof(uint32);
+ char buf[sizeof(uint32)];
+
+ memset(buf, 0, buflen);
+ XLogWalRcvWrite(buf, buflen, LogstreamResult.Write, tli);
+ }
+
if (LogstreamResult.Flush < LogstreamResult.Write)
{
WalRcvData *walrcv = WalRcv;
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-07-17 00:49 Noah Misch <noah@leadboat.com>
parent: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
0 siblings, 2 replies; 86+ messages in thread
From: Noah Misch @ 2023-07-17 00:49 UTC (permalink / raw)
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; +Cc: thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Tue, Jul 11, 2023 at 03:54:14PM +0900, Kyotaro Horiguchi wrote:
> At Mon, 10 Jul 2023 13:00:12 -0700, Noah Misch <noah@leadboat.com> wrote in
> > On Mon, May 15, 2023 at 03:38:17PM +1200, Thomas Munro wrote:
> > > On Fri, May 12, 2023 at 6:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> > > > 2023-05-11 20:19:22.248 MSK [2037134] FATAL: invalid memory alloc request size 2021163525
> > > > 2023-05-11 20:19:22.248 MSK [2037114] LOG: startup process (PID 2037134) exited with exit code 1
On my system, testing unpatched REL_15_2, the test reached its first failure
at minute 53 and its second failure at minute 189. Perhaps a faster,
deterministic test would be feasible:
- Use pg_logical_emit_message to fill a few segments with 0xFF.
- CHECKPOINT the primary, so the standby recycles segments.
- One more pg_logical_emit_message, computing the length from
pg_current_wal_insert_lsn() such that new message crosses a segment boundary
and ends 4 bytes before the end of a page.
- Stop the primary.
- If the bug is present, the standby will exit.
> > > Thanks Alexander. Looking into this. I think it is probably
> > > something like: recycled standby pages are not zeroed (something we
> > > already needed to do something about[1]), and when we read a recycled
> > > garbage size (like your "xxxx") at the end of a page at an offset
> > > where we don't have a full record header on one page, we skip the
> > > ValidXLogRecordHeader() call (and always did), but the check in
> > > allocate_recordbuf() which previously handled that "gracefully" (well,
> > > it would try to allocate up to 1GB bogusly, but it wouldn't try to
> > > allocate more than that and ereport) is a bit too late. I probably
> > > need to add an earlier not-too-big validation. Thinking.
> >
> > I agree about an earlier not-too-big validation. Like the attached? I
> > haven't tested it with Alexander's recipe or pondered it thoroughly.
>
> I like the patch for its code clean up, but I'm afraid that it removes
> the existing record length check when reading continued
> pages.
The removed check would be repetitive in all cases known to me, but I wouldn't
mind keeping the check instead.
> However, I'm unsure about merely adding a check for too-long
> records, due to the potential for requesting an excessively large
> amount of memory, even if it will be released shortly.
>
> > > [1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
> >
> > Regarding [1], is it still worth zeroing recycled pages on standbys and/or
> > reading the whole header before allocating xl_tot_len? (Are there benefits
> > other than avoiding a 1G backend allocation or 4G frontend allocation, or is
> > that benefit worth the cycles?)
>
> I believe reading the whole header is the most sensible approach as it
> can prevent unnecessary memory requests. Another potential solution
> (or hack) for this specific case is to let XLogWalRcvFlush write a
> finalizing ((uint32)0) when dying is true. This stabilizes the
> behavior to "invalid record length.. got 0" when running the TAP test.
> --- a/src/backend/replication/walreceiver.c
> +++ b/src/backend/replication/walreceiver.c
> @@ -989,6 +989,15 @@ XLogWalRcvFlush(bool dying, TimeLineID tli)
> {
> Assert(tli != 0);
>
> + if (dying)
> + {
> + int buflen = sizeof(uint32);
> + char buf[sizeof(uint32)];
> +
> + memset(buf, 0, buflen);
> + XLogWalRcvWrite(buf, buflen, LogstreamResult.Write, tli);
> + }
> +
> if (LogstreamResult.Flush < LogstreamResult.Write)
> {
> WalRcvData *walrcv = WalRcv;
That's inexpensive in cycles, and it should make the failures in question
quite rare. If we do that instead of pre-zeroing, I think we'd still want to
be ready for invalid lengths, for the case of unclean standby shutdown.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-07-20 06:01 Kyotaro Horiguchi <horikyota.ntt@gmail.com>
parent: Noah Misch <noah@leadboat.com>
1 sibling, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2023-07-20 06:01 UTC (permalink / raw)
To: noah@leadboat.com; +Cc: thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
At Sun, 16 Jul 2023 17:49:05 -0700, Noah Misch <noah@leadboat.com> wrote in
> On Tue, Jul 11, 2023 at 03:54:14PM +0900, Kyotaro Horiguchi wrote:
> > I like the patch for its code clean up, but I'm afraid that it removes
> > the existing record length check when reading continued
> > pages.
>
> The removed check would be repetitive in all cases known to me, but I wouldn't
> mind keeping the check instead.
Ah. I got it. No need to do that again.
> > I believe reading the whole header is the most sensible approach as it
> > can prevent unnecessary memory requests. Another potential solution
> > (or hack) for this specific case is to let XLogWalRcvFlush write a
> > finalizing ((uint32)0) when dying is true. This stabilizes the
> > behavior to "invalid record length.. got 0" when running the TAP test.
...
> That's inexpensive in cycles, and it should make the failures in question
> quite rare. If we do that instead of pre-zeroing, I think we'd still want to
> be ready for invalid lengths, for the case of unclean standby shutdown.
Agreed.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-10 07:45 Michael Paquier <michael@paquier.xyz>
parent: Noah Misch <noah@leadboat.com>
1 sibling, 2 replies; 86+ messages in thread
From: Michael Paquier @ 2023-08-10 07:45 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sun, Jul 16, 2023 at 05:49:05PM -0700, Noah Misch wrote:
> On my system, testing unpatched REL_15_2, the test reached its first failure
> at minute 53 and its second failure at minute 189. Perhaps a faster,
> deterministic test would be feasible:
It took less than five minutes for me to reproduce the failure using
the test case sent by Alexander on HEAD:
2023-08-10 15:26:37.401 JST [11239] FATAL: invalid memory alloc request size 2021163525
2023-08-10 15:26:37.405 JST [11235] LOG: startup process (PID 11239) exited with exit code 1
I don't see it after two hours of tests with the patch.
> - Use pg_logical_emit_message to fill a few segments with 0xFF.
> - CHECKPOINT the primary, so the standby recycles segments.
> - One more pg_logical_emit_message, computing the length from
> pg_current_wal_insert_lsn() such that new message crosses a segment boundary
> and ends 4 bytes before the end of a page.
> - Stop the primary.
> - If the bug is present, the standby will exit.
Good idea to pollute the data with recycled segments. Using a minimal
WAL segment size whould help here as well in keeping a test cheap, and
two segments should be enough. The alignment calculations and the
header size can be known, but the standby records are an issue for the
predictability of the test when it comes to adjust the length of the
logical message depending on the 8k WAL page, no?
>> However, I'm unsure about merely adding a check for too-long
>> records, due to the potential for requesting an excessively large
>> amount of memory, even if it will be released shortly.
>>
>>>> [1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
>>>
>>> Regarding [1], is it still worth zeroing recycled pages on standbys and/or
>>> reading the whole header before allocating xl_tot_len? (Are there benefits
>>> other than avoiding a 1G backend allocation or 4G frontend allocation, or is
>>> that benefit worth the cycles?)
>>
>> I believe reading the whole header is the most sensible approach as it
>> can prevent unnecessary memory requests. Another potential solution
>> (or hack) for this specific case is to let XLogWalRcvFlush write a
>> finalizing ((uint32)0) when dying is true. This stabilizes the
>> behavior to "invalid record length.. got 0" when running the TAP test.
FWIW, I came back to this thread while tweaking the error reporting of
xlogreader.c for the sake of this thread and this proposal is an
improvement to be able to make a distinction between an OOM and an
incorrect record:
https://www.postgresql.org/message-id/ZMh/WV+CuknqePQQ@paquier.xyz
Anyway, agreed that it's an improvement to remove this check out of
allocate_recordbuf(). Noah, are you planning to work more on that?
>> --- a/src/backend/replication/walreceiver.c
>> +++ b/src/backend/replication/walreceiver.c
>> @@ -989,6 +989,15 @@ XLogWalRcvFlush(bool dying, TimeLineID tli)
>> {
>> Assert(tli != 0);
>>
>> + if (dying)
>> + {
>> + int buflen = sizeof(uint32);
>> + char buf[sizeof(uint32)];
>> +
>> + memset(buf, 0, buflen);
>> + XLogWalRcvWrite(buf, buflen, LogstreamResult.Write, tli);
>> + }
>> +
>> if (LogstreamResult.Flush < LogstreamResult.Write)
>> {
>> WalRcvData *walrcv = WalRcv;
>
> That's inexpensive in cycles, and it should make the failures in question
> quite rare. If we do that instead of pre-zeroing, I think we'd still want to
> be ready for invalid lengths, for the case of unclean standby shutdown.
Tweaking the WAL receiver to do more zeroing has been discussed in the
past, but as far as I recall we were not completely sure whether it's
completely free, either. It seems to me that this should have a
discussion on its own, in a new thread. Not sure if we should worry
about archiving, actually, even if the segments should be padded with
zeros beforehand.
+ gotheader = targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord ? true : false;
Perhaps this could use more parenthesis around the expression checked,
for readability.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNSVlR%2Fv8gTDIMsU@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-10 23:22 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2023-08-10 23:22 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Thu, Aug 10, 2023 at 04:45:25PM +0900, Michael Paquier wrote:
> On Sun, Jul 16, 2023 at 05:49:05PM -0700, Noah Misch wrote:
>> - Use pg_logical_emit_message to fill a few segments with 0xFF.
>> - CHECKPOINT the primary, so the standby recycles segments.
>> - One more pg_logical_emit_message, computing the length from
>> pg_current_wal_insert_lsn() such that new message crosses a segment boundary
>> and ends 4 bytes before the end of a page.
>> - Stop the primary.
>> - If the bug is present, the standby will exit.
>
> Good idea to pollute the data with recycled segments. Using a minimal
> WAL segment size whould help here as well in keeping a test cheap, and
> two segments should be enough. The alignment calculations and the
> header size can be known, but the standby records are an issue for the
> predictability of the test when it comes to adjust the length of the
> logical message depending on the 8k WAL page, no?
Actually, for this one, I think that I have a simpler idea to make it
deterministic. Once we have inserted a record at the page limit on
the primary, we can:
- Stop the standby
- Stop the primary
- Rewrite by ourselves a few bytes in the last segment on the standby
to emulate a recycled segment portion, based on the end LSN of the
logical message record, retrieved either with pg_walinspect or
pg_waldump.
- Start the standby, which would replay up to the previous record at
the page limit.
- The standby just be in a state where it waits for the missing
records from the primary and keeps looking at streaming, but it should
not fail startup.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNVxSMWZdNOXN9sH@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-11 02:58 Noah Misch <noah@leadboat.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 86+ messages in thread
From: Noah Misch @ 2023-08-11 02:58 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Thu, Aug 10, 2023 at 04:45:25PM +0900, Michael Paquier wrote:
> On Sun, Jul 16, 2023 at 05:49:05PM -0700, Noah Misch wrote:
> > On my system, testing unpatched REL_15_2, the test reached its first failure
> > at minute 53 and its second failure at minute 189. Perhaps a faster,
> > deterministic test would be feasible:
>
> It took less than five minutes for me to reproduce the failure using
> the test case sent by Alexander on HEAD:
> 2023-08-10 15:26:37.401 JST [11239] FATAL: invalid memory alloc request size 2021163525
> 2023-08-10 15:26:37.405 JST [11235] LOG: startup process (PID 11239) exited with exit code 1
>
> I don't see it after two hours of tests with the patch.
That is good.
> > - Use pg_logical_emit_message to fill a few segments with 0xFF.
> > - CHECKPOINT the primary, so the standby recycles segments.
> > - One more pg_logical_emit_message, computing the length from
> > pg_current_wal_insert_lsn() such that new message crosses a segment boundary
> > and ends 4 bytes before the end of a page.
> > - Stop the primary.
> > - If the bug is present, the standby will exit.
>
> Good idea to pollute the data with recycled segments. Using a minimal
> WAL segment size whould help here as well in keeping a test cheap, and
> two segments should be enough. The alignment calculations and the
> header size can be known, but the standby records are an issue for the
> predictability of the test when it comes to adjust the length of the
> logical message depending on the 8k WAL page, no?
Could be. I expect there would be challenges translating that outline into a
real test, but I don't know if that will be a major one. The test doesn't
need to be 100% deterministic. If it fails 25% of the time and is not the
slowest test in the recovery suite, I'd find that good enough.
> >>>> [1] https://www.postgresql.org/message-id/20210505010835.umylslxgq4a6rbwg@alap3.anarazel.de
> >>>
> >>> Regarding [1], is it still worth zeroing recycled pages on standbys and/or
> >>> reading the whole header before allocating xl_tot_len? (Are there benefits
> >>> other than avoiding a 1G backend allocation or 4G frontend allocation, or is
> >>> that benefit worth the cycles?)
> >>
> >> I believe reading the whole header is the most sensible approach as it
> >> can prevent unnecessary memory requests. Another potential solution
> >> (or hack) for this specific case is to let XLogWalRcvFlush write a
> >> finalizing ((uint32)0) when dying is true. This stabilizes the
> >> behavior to "invalid record length.. got 0" when running the TAP test.
>
> FWIW, I came back to this thread while tweaking the error reporting of
> xlogreader.c for the sake of this thread and this proposal is an
> improvement to be able to make a distinction between an OOM and an
> incorrect record:
> https://www.postgresql.org/message-id/ZMh/WV+CuknqePQQ@paquier.xyz
>
> Anyway, agreed that it's an improvement to remove this check out of
> allocate_recordbuf(). Noah, are you planning to work more on that?
I can push xl_tot_len-validate-v1.patch, particularly given the testing result
you reported today. I'm content for my part to stop there.
> >> --- a/src/backend/replication/walreceiver.c
> >> +++ b/src/backend/replication/walreceiver.c
> >> @@ -989,6 +989,15 @@ XLogWalRcvFlush(bool dying, TimeLineID tli)
> >> {
> >> Assert(tli != 0);
> >>
> >> + if (dying)
> >> + {
> >> + int buflen = sizeof(uint32);
> >> + char buf[sizeof(uint32)];
> >> +
> >> + memset(buf, 0, buflen);
> >> + XLogWalRcvWrite(buf, buflen, LogstreamResult.Write, tli);
> >> + }
> >> +
> >> if (LogstreamResult.Flush < LogstreamResult.Write)
> >> {
> >> WalRcvData *walrcv = WalRcv;
> >
> > That's inexpensive in cycles, and it should make the failures in question
> > quite rare. If we do that instead of pre-zeroing, I think we'd still want to
> > be ready for invalid lengths, for the case of unclean standby shutdown.
>
> Tweaking the WAL receiver to do more zeroing has been discussed in the
> past, but as far as I recall we were not completely sure whether it's
> completely free, either. It seems to me that this should have a
> discussion on its own, in a new thread. Not sure if we should worry
> about archiving, actually, even if the segments should be padded with
> zeros beforehand.
Okay.
> + gotheader = targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord ? true : false;
Okay.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-11 06:08 Michael Paquier <michael@paquier.xyz>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-11 06:08 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Thu, Aug 10, 2023 at 07:58:08PM -0700, Noah Misch wrote:
> On Thu, Aug 10, 2023 at 04:45:25PM +0900, Michael Paquier wrote:
>> Good idea to pollute the data with recycled segments. Using a minimal
>> WAL segment size whould help here as well in keeping a test cheap, and
>> two segments should be enough. The alignment calculations and the
>> header size can be known, but the standby records are an issue for the
>> predictability of the test when it comes to adjust the length of the
>> logical message depending on the 8k WAL page, no?
>
> Could be. I expect there would be challenges translating that outline into a
> real test, but I don't know if that will be a major one. The test doesn't
> need to be 100% deterministic. If it fails 25% of the time and is not the
> slowest test in the recovery suite, I'd find that good enough.
FWIW, I'm having a pretty hard time to get something close enough to a
page border in a reliable way. Perhaps using a larger series of
records and select only one would be more reliable.. Need to test
that a bit more.
>> FWIW, I came back to this thread while tweaking the error reporting of
>> xlogreader.c for the sake of this thread and this proposal is an
>> improvement to be able to make a distinction between an OOM and an
>> incorrect record:
>> https://www.postgresql.org/message-id/ZMh/WV+CuknqePQQ@paquier.xyz
>>
>> Anyway, agreed that it's an improvement to remove this check out of
>> allocate_recordbuf(). Noah, are you planning to work more on that?
>
> I can push xl_tot_len-validate-v1.patch, particularly given the testing result
> you reported today. I'm content for my part to stop there.
Okay, fine by me. That's going to help with what I am doing in the
other thread as I'd need to make a better difference between the OOM
and the invalid cases for the allocation path.
You are planning for a backpatch to take care of the inconsistency,
right? The report has been on 15~ where the prefetching was
introduced. I'd be OK to just do that and not mess up with the stable
branches more than necessary (aka ~14) if nobody complains, especially
REL_11_STABLE planned to be EOL'd in the next minor cycle.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNXQWvnmt1Fpt6vu@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-11 14:00 Noah Misch <noah@leadboat.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Noah Misch @ 2023-08-11 14:00 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Fri, Aug 11, 2023 at 03:08:26PM +0900, Michael Paquier wrote:
> On Thu, Aug 10, 2023 at 07:58:08PM -0700, Noah Misch wrote:
> > On Thu, Aug 10, 2023 at 04:45:25PM +0900, Michael Paquier wrote:
> >> Good idea to pollute the data with recycled segments. Using a minimal
> >> WAL segment size whould help here as well in keeping a test cheap, and
> >> two segments should be enough. The alignment calculations and the
> >> header size can be known, but the standby records are an issue for the
> >> predictability of the test when it comes to adjust the length of the
> >> logical message depending on the 8k WAL page, no?
> >
> > Could be. I expect there would be challenges translating that outline into a
> > real test, but I don't know if that will be a major one. The test doesn't
> > need to be 100% deterministic. If it fails 25% of the time and is not the
> > slowest test in the recovery suite, I'd find that good enough.
>
> FWIW, I'm having a pretty hard time to get something close enough to a
> page border in a reliable way. Perhaps using a larger series of
> records and select only one would be more reliable.. Need to test
> that a bit more.
Interesting. So pg_logical_emit_message(false, 'X', repeat('X', n)) doesn't
get close enough, but s/n/n+1/ spills to the next page? If so, I did not
anticipate that.
> >> FWIW, I came back to this thread while tweaking the error reporting of
> >> xlogreader.c for the sake of this thread and this proposal is an
> >> improvement to be able to make a distinction between an OOM and an
> >> incorrect record:
> >> https://www.postgresql.org/message-id/ZMh/WV+CuknqePQQ@paquier.xyz
> >>
> >> Anyway, agreed that it's an improvement to remove this check out of
> >> allocate_recordbuf(). Noah, are you planning to work more on that?
> >
> > I can push xl_tot_len-validate-v1.patch, particularly given the testing result
> > you reported today. I'm content for my part to stop there.
>
> Okay, fine by me. That's going to help with what I am doing in the
> other thread as I'd need to make a better difference between the OOM
> and the invalid cases for the allocation path.
>
> You are planning for a backpatch to take care of the inconsistency,
> right? The report has been on 15~ where the prefetching was
> introduced. I'd be OK to just do that and not mess up with the stable
> branches more than necessary (aka ~14) if nobody complains, especially
> REL_11_STABLE planned to be EOL'd in the next minor cycle.
I recall earlier messages theorizing that it was just harder to hit in v14, so
I'm disinclined to stop at v15. I think the main choice is whether to stop at
v11 (normal choice) or v12 (worry about breaking the last v11 point release).
I don't have a strong opinion between those.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-11 23:56 Thomas Munro <thomas.munro@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-11 23:56 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sat, Aug 12, 2023 at 2:00 AM Noah Misch <noah@leadboat.com> wrote:
> On Fri, Aug 11, 2023 at 03:08:26PM +0900, Michael Paquier wrote:
> > You are planning for a backpatch to take care of the inconsistency,
> > right? The report has been on 15~ where the prefetching was
> > introduced. I'd be OK to just do that and not mess up with the stable
> > branches more than necessary (aka ~14) if nobody complains, especially
> > REL_11_STABLE planned to be EOL'd in the next minor cycle.
>
> I recall earlier messages theorizing that it was just harder to hit in v14, so
> I'm disinclined to stop at v15. I think the main choice is whether to stop at
> v11 (normal choice) or v12 (worry about breaking the last v11 point release).
> I don't have a strong opinion between those.
Thanks for working on this.
I wonder if we need a more explicit way to construct pages with the
right bits to reach interesting test cases and get full enough
coverage... (Cf throwing SQL at the WAL to see if it sticks.)
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-12 00:20 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-12 00:20 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sat, Aug 12, 2023 at 11:56:45AM +1200, Thomas Munro wrote:
> On Sat, Aug 12, 2023 at 2:00 AM Noah Misch <noah@leadboat.com> wrote:
>> On Fri, Aug 11, 2023 at 03:08:26PM +0900, Michael Paquier wrote:
>> I recall earlier messages theorizing that it was just harder to hit in v14, so
>> I'm disinclined to stop at v15. I think the main choice is whether to stop at
>> v11 (normal choice) or v12 (worry about breaking the last v11 point release).
>> I don't have a strong opinion between those.
Okay. I wouldn't be inclined to patch v11 for that, FWIW, as this
code path is touched by recovery and more. At least it does not seem
worth taking any risk compared to the potential gain.
> Thanks for working on this.
>
> I wonder if we need a more explicit way to construct pages with the
> right bits to reach interesting test cases and get full enough
> coverage... (Cf throwing SQL at the WAL to see if it sticks.)
You mean SQL functions that write an arbitrary set of bytes at a given
LSN, to trigger some expected behavior on a follow-up crash recovery?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNbQUDcZK0ELn1o9@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-12 00:32 Tom Lane <tgl@sss.pgh.pa.us>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-08-12 00:32 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
Michael Paquier <michael@paquier.xyz> writes:
> On Sat, Aug 12, 2023 at 11:56:45AM +1200, Thomas Munro wrote:
>> On Sat, Aug 12, 2023 at 2:00 AM Noah Misch <noah@leadboat.com> wrote:
>>> I recall earlier messages theorizing that it was just harder to hit in v14, so
>>> I'm disinclined to stop at v15. I think the main choice is whether to stop at
>>> v11 (normal choice) or v12 (worry about breaking the last v11 point release).
>>> I don't have a strong opinion between those.
> Okay. I wouldn't be inclined to patch v11 for that, FWIW, as this
> code path is touched by recovery and more. At least it does not seem
> worth taking any risk compared to the potential gain.
That was my gut reaction too -- risk/reward seems not great for the
last v11 release. (I've been burned a couple of times now by putting
can't-fix-it-anymore bugs into the final release of a branch, so maybe
I'm just being overly paranoid. But it's something to worry about.)
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-12 21:13 Noah Misch <noah@leadboat.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 86+ messages in thread
From: Noah Misch @ 2023-08-12 21:13 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Thomas Munro <thomas.munro@gmail.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Fri, Aug 11, 2023 at 08:32:00PM -0400, Tom Lane wrote:
> Michael Paquier <michael@paquier.xyz> writes:
> > On Sat, Aug 12, 2023 at 11:56:45AM +1200, Thomas Munro wrote:
> >> On Sat, Aug 12, 2023 at 2:00 AM Noah Misch <noah@leadboat.com> wrote:
> >>> I recall earlier messages theorizing that it was just harder to hit in v14, so
> >>> I'm disinclined to stop at v15. I think the main choice is whether to stop at
> >>> v11 (normal choice) or v12 (worry about breaking the last v11 point release).
> >>> I don't have a strong opinion between those.
>
> > Okay. I wouldn't be inclined to patch v11 for that, FWIW, as this
> > code path is touched by recovery and more. At least it does not seem
> > worth taking any risk compared to the potential gain.
>
> That was my gut reaction too -- risk/reward seems not great for the
> last v11 release. (I've been burned a couple of times now by putting
> can't-fix-it-anymore bugs into the final release of a branch, so maybe
> I'm just being overly paranoid. But it's something to worry about.)
Okay, v12+ would be fine with me. Alas, a closer look at v1 found two
defects, one with severity far greater than the bug v1 aimed to fix:
===== 1. For oversized records, it changed startup FATAL to silent data loss
This affects only supported branches (v15-). In v16+, commit 8fcb32d
introduced and checked XLogRecordMaxSize. But in v15 with patch v1, redo of
pg_logical_emit_message(false, '_', repeat('x', 1024 * 1024 * 1024 - 1000))
just ends at the oversized record:
2523581 2023-08-12 17:17:39.512 GMT LOG: redo starts at 0/41F155B8
2523581 2023-08-12 17:17:39.512 GMT LOG: record length 1073740879 at 0/41F155F0 too long
2523581 2023-08-12 17:17:39.512 GMT LOG: redo done at 0/41F155B8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
Any user could call pg_logical_emit_message() to silently terminate the WAL
stream, which is far worse than the original bug. So far, I'm seeing one way
to clearly fix $SUBJECT without that harm. When a record header spans a page
boundary, read the next page to reassemble the header. If
!ValidXLogRecordHeader() (invalid xl_rmid or xl_prev), treat as end of WAL.
Otherwise, read the whole record in chunks, calculating CRC. If CRC is
invalid, treat as end of WAL. Otherwise, ereport(FATAL) for the oversized
record. A side benefit would be avoiding useless large allocations (1GB
backend, 4GB frontend) as discussed upthread. May as well do the xl_rmid and
xl_prev checks in all branches, to avoid needless XLogRecordMaxSize-1
allocations. Thoughts? For invalid-length records in v16+, since every such
record is end-of-wal or corruption, those versions could skip the CRC.
As an alternative, zeroing recycled pages could remove "most of" the spurious
errors without adding silent data loss. I considered another alternative of
back-patching XLogRecordMaxSize and proceeding with a patch like v1, but that
wouldn't help with records in existing WAL archives. Of course, we could also
choose to leave $SUBJECT unfixed in v15-.
===== 2. DecodeXLogRecordRequiredSpace() is the wrong ceiling
v1 treats DecodeXLogRecordRequiredSpace(L) (=L+2445 here) as the max needed,
but allocate_recordbuf(state, L) allocates up to L+8192. If the uninitialized
length happened to contain an integer between about 1<<30-8192 and 1<<30-2445,
one still got e.g. "FATAL: invalid memory alloc request size 1073741824".
Since v16+ has XLogRecordMaxSize, testing XLogRecordMaxSize avoids the problem
and should suffice:
===
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -640,7 +640,6 @@ restart:
(uint32) SizeOfXLogRecord, total_len);
goto err;
}
-#ifndef FRONTEND
/*
* We may be looking at a random uint32 read from a recycled segment. For
@@ -651,13 +650,12 @@ restart:
* the allocation may succeed but record checks are going to fail so this
* would be short-lived.
*/
- if (!AllocSizeIsValid(DecodeXLogRecordRequiredSpace(total_len)))
+ if (total_len > XLogRecordMaxSize)
{
report_invalid_record(state, "record length %u at %X/%X too long",
total_len, LSN_FORMAT_ARGS(RecPtr));
goto err;
}
-#endif
/*
* If the whole record header is on this page, validate it immediately.
===
allocate_recordbuf()'s addend is not critical; it's just to "avoid useless
small increases". v15- should avoid the problem like this:
===
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 969bcc3..1bac303 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -201,4 +201,8 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
+#ifndef FRONTEND
+ if (!AllocSizeIsValid(newSize))
+ newSize = reclength;
+#endif
if (state->readRecordBuf)
===
In v15, one can reach this case with pg_logical_emit_message(false, '_',
repeat('x', 1024 * 1024 * 1024 - 4000)). On v16+, one might reach this with
unlucky trailing garbage, but XLogRecordMaxSize prevents reaching it through
pg_logical_emit_message().
Thanks,
nm
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-13 03:12 Thomas Munro <thomas.munro@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-13 03:12 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sun, Aug 13, 2023 at 9:13 AM Noah Misch <noah@leadboat.com> wrote:
> Any user could call pg_logical_emit_message() to silently terminate the WAL
> stream, which is far worse than the original bug. So far, I'm seeing one way
> to clearly fix $SUBJECT without that harm. When a record header spans a page
> boundary, read the next page to reassemble the header. If
> !ValidXLogRecordHeader() (invalid xl_rmid or xl_prev), treat as end of WAL.
> Otherwise, read the whole record in chunks, calculating CRC. If CRC is
> invalid, treat as end of WAL. Otherwise, ereport(FATAL) for the oversized
> record. A side benefit would be avoiding useless large allocations (1GB
> backend, 4GB frontend) as discussed upthread. May as well do the xl_rmid and
> xl_prev checks in all branches, to avoid needless XLogRecordMaxSize-1
> allocations. Thoughts? For invalid-length records in v16+, since every such
> record is end-of-wal or corruption, those versions could skip the CRC.
That sounds quite strong. But... why couldn't the existing
xlp_rem_len cross-check protect us from this failure mode? If we
could defer the allocation until after that check (and the usual
ValidXLogRecordHeader() check), I think we'd know that we're really
looking at a size that was actually written in both pages (which must
also have survived xlp_pageaddr check), no?
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-13 03:30 Noah Misch <noah@leadboat.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Noah Misch @ 2023-08-13 03:30 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sun, Aug 13, 2023 at 03:12:34PM +1200, Thomas Munro wrote:
> On Sun, Aug 13, 2023 at 9:13 AM Noah Misch <noah@leadboat.com> wrote:
> > Any user could call pg_logical_emit_message() to silently terminate the WAL
> > stream, which is far worse than the original bug. So far, I'm seeing one way
> > to clearly fix $SUBJECT without that harm. When a record header spans a page
> > boundary, read the next page to reassemble the header. If
> > !ValidXLogRecordHeader() (invalid xl_rmid or xl_prev), treat as end of WAL.
> > Otherwise, read the whole record in chunks, calculating CRC. If CRC is
> > invalid, treat as end of WAL. Otherwise, ereport(FATAL) for the oversized
> > record. A side benefit would be avoiding useless large allocations (1GB
> > backend, 4GB frontend) as discussed upthread. May as well do the xl_rmid and
> > xl_prev checks in all branches, to avoid needless XLogRecordMaxSize-1
> > allocations. Thoughts? For invalid-length records in v16+, since every such
> > record is end-of-wal or corruption, those versions could skip the CRC.
>
> That sounds quite strong. But... why couldn't the existing
> xlp_rem_len cross-check protect us from this failure mode? If we
> could defer the allocation until after that check (and the usual
> ValidXLogRecordHeader() check), I think we'd know that we're really
> looking at a size that was actually written in both pages (which must
> also have survived xlp_pageaddr check), no?
Hmm, I think that is right. A coincidental match of the 32-bit CRC is more
probable than having all those fields appear valid by coincidence, especially
xlp_pageaddr.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-14 02:08 Michael Paquier <michael@paquier.xyz>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-14 02:08 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Sat, Aug 12, 2023 at 08:30:34PM -0700, Noah Misch wrote:
> On Sun, Aug 13, 2023 at 03:12:34PM +1200, Thomas Munro wrote:
>> On Sun, Aug 13, 2023 at 9:13 AM Noah Misch <noah@leadboat.com> wrote:
>>> Any user could call pg_logical_emit_message() to silently terminate the WAL
>>> stream, which is far worse than the original bug. So far, I'm seeing one way
>>> to clearly fix $SUBJECT without that harm. When a record header spans a page
>>> boundary, read the next page to reassemble the header. If
>>> !ValidXLogRecordHeader() (invalid xl_rmid or xl_prev), treat as end of WAL.
>>> Otherwise, read the whole record in chunks, calculating CRC. If CRC is
>>> invalid, treat as end of WAL. Otherwise, ereport(FATAL) for the oversized
>>> record. A side benefit would be avoiding useless large allocations (1GB
>>> backend, 4GB frontend) as discussed upthread. May as well do the xl_rmid and
>>> xl_prev checks in all branches, to avoid needless XLogRecordMaxSize-1
>>> allocations. Thoughts? For invalid-length records in v16+, since every such
>>> record is end-of-wal or corruption, those versions could skip the CRC.
>>
>> That sounds quite strong. But... why couldn't the existing
>> xlp_rem_len cross-check protect us from this failure mode? If we
>> could defer the allocation until after that check (and the usual
>> ValidXLogRecordHeader() check), I think we'd know that we're really
>> looking at a size that was actually written in both pages (which must
>> also have survived xlp_pageaddr check), no?
+1 for relying on xlp_rem_len for that.
> Hmm, I think that is right. A coincidental match of the 32-bit CRC is more
> probable than having all those fields appear valid by coincidence, especially
> xlp_pageaddr.
Hmm. [.. thinks ..]
It seems to me that we should try to also finish the header validation
before attempting XLogReadRecordAlloc() on the total_len as well? It
looks like the end result would be to move the first ReadPageInternal
done for the header with all its cross-page checks before
XLogReadRecordAlloc(). That should remove the need of having
gotheader from v1.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNmMqEXJ6ECTQ0Ys@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-14 03:56 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-14 03:56 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Aug 14, 2023 at 2:08 PM Michael Paquier <michael@paquier.xyz> wrote:
> It seems to me that we should try to also finish the header validation
> before attempting XLogReadRecordAlloc() on the total_len as well? It
> looks like the end result would be to move the first ReadPageInternal
> done for the header with all its cross-page checks before
> XLogReadRecordAlloc(). That should remove the need of having
> gotheader from v1.
Yeah. Trying out an idea for how to untangle that...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-14 10:28 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-14 10:28 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Aug 14, 2023 at 3:56 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Mon, Aug 14, 2023 at 2:08 PM Michael Paquier <michael@paquier.xyz> wrote:
> > It seems to me that we should try to also finish the header validation
> > before attempting XLogReadRecordAlloc() on the total_len as well? It
> > looks like the end result would be to move the first ReadPageInternal
> > done for the header with all its cross-page checks before
> > XLogReadRecordAlloc(). That should remove the need of having
> > gotheader from v1.
>
> Yeah. Trying out an idea for how to untangle that...
Something like this... patches for 12 and master attached. Needs a
lot more testing. I really want to figure out how to get
deterministic tests...
Attachments:
[text/x-patch] 0001-Don-t-trust-unvalidated-xl_tot_len-v12.patch (5.5K, ../../CA+hUKGLOgGh=9siEhACUsrZEAQPVEQWgn35cggeci0_cNbPSSw@mail.gmail.com/2-0001-Don-t-trust-unvalidated-xl_tot_len-v12.patch)
download | inline diff:
From f77f066503caf745c3832c30e92c9a96ce798d8e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 21:28:24 +1200
Subject: [PATCH] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally in old and
new branches using pre-existing page-level validation.
Whenever a record header spans a page boundary, we just need to defer
all memory allocation based on xl_tot_len until we've read the following
page. At that point we know that its first sector is not recycled data,
by checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 12. This failure mode is rare, and 11 is about to go out
of support, so we decided not to change its final release.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 64 +++++++++++++------------
1 file changed, 33 insertions(+), 31 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 26f8e0ae2c..697965c749 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -166,18 +166,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
@@ -346,15 +334,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -370,17 +350,10 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -475,6 +448,35 @@ restart:
goto err;
gotheader = true;
}
+
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
} while (gotlen < total_len);
Assert(gotheader);
--
2.41.0
[text/x-patch] 0001-Don-t-trust-unvalidated-xl_tot_len-master.patch (7.8K, ../../CA+hUKGLOgGh=9siEhACUsrZEAQPVEQWgn35cggeci0_cNbPSSw@mail.gmail.com/3-0001-Don-t-trust-unvalidated-xl_tot_len-master.patch)
download | inline diff:
From 43b85fb6877cec3b9227a9ee808c44e3acdeb1d2 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally in old and
new branches using pre-existing page-level validation.
Whenever a record header spans a page boundary, we just need to defer
all memory allocation based on xl_tot_len until we've read the following
page. At that point we know that its first sector is not recycled data,
by checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 12. This failure mode is rare, and 11 is about to go out
of support, so we decided not to change its final release.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 110 +++++++++++++-----------
1 file changed, 61 insertions(+), 49 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..d5ed139131 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -203,18 +203,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
@@ -669,41 +657,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +691,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +792,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +862,25 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /* We failed to allocate memory for an oversized record. */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +909,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
--
2.41.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-15 02:04 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-15 02:04 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Aug 14, 2023 at 10:28:54PM +1200, Thomas Munro wrote:
> Something like this... patches for 12 and master attached. Needs a
> lot more testing. I really want to figure out how to get
> deterministic tests...
Interesting, thanks for the quick patch!
/*
* Try to find space to decode this record, if we can do so without
* calling palloc. If we can't, we'll try again below after we've
* validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
false /* allow_oversized */ );
The patch relies on total_len before the header validation is
completed, meaning that this value of total_len could be invalid
because it comes from the partially-read header.. Oh wait, that's
actually OK because an oversized allocation is not allowed yet and the
decode buffer would not be able to store more than what it can?
Perhaps this comment should be updated to tell something like, adding
that total_len can be garbage, but we don't care because we don't
allocate anything that the decode buffer cannot hold.
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
Wouldn't it be OK to drop entirely this check? I'm fine to keep it as
a safety measure, but it should not be necessary now that it gets
called only once the header is validated?
Regarding the tests, I have been using this formula to produce the
number of bytes until the next page boundary:
select setting::int - ((pg_current_wal_lsn() - '0/0') % setting::int)
from pg_settings where name = 'wal_block_size';
Using pg_logical_emit_message() non-transactional with an empty set of
strings generates records of 56 bytes, so I was thinking about the
following:
- Generate a bunch of small records with pg_logical_emit_message(), or
records based on the threshold with the previous formula.
- Loop until we reach a page limit, at 24 bytes (?).
- Generate one last record to cut through.
- Stop the node in immediate mode.
- Write some garbage bytes on the last page generated to emulate the
recycled contents and an allocation
- Start the node, which should be able to startup.
With wal_level = minimal, autovacuum = off and a large
checkpoint_timeout, any other records are minimized. That's fancy,
though.
Do you think that this could work reliably?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNrdQSWe1flOfHN3@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-15 02:36 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-15 02:36 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Tue, Aug 15, 2023 at 2:05 PM Michael Paquier <michael@paquier.xyz> wrote:
> /*
> * Try to find space to decode this record, if we can do so without
> * calling palloc. If we can't, we'll try again below after we've
> * validated that total_len isn't garbage bytes from a recycled WAL page.
> */
> decoded = XLogReadRecordAlloc(state,
> total_len,
> false /* allow_oversized */ );
>
> The patch relies on total_len before the header validation is
> completed, meaning that this value of total_len could be invalid
> because it comes from the partially-read header.. Oh wait, that's
> actually OK because an oversized allocation is not allowed yet and the
> decode buffer would not be able to store more than what it can?
> Perhaps this comment should be updated to tell something like, adding
> that total_len can be garbage, but we don't care because we don't
> allocate anything that the decode buffer cannot hold.
Yeah.
> #ifndef FRONTEND
>
> - /*
> - * Note that in much unlucky circumstances, the random data read from a
> - * recycled segment can cause this routine to be called with a size
> - * causing a hard failure at allocation. For a standby, this would cause
> - * the instance to stop suddenly with a hard failure, preventing it to
> - * retry fetching WAL from one of its sources which could allow it to move
> - * on with replay without a manual restart. If the data comes from a past
> - * recycled segment and is still valid, then the allocation may succeed
> - * but record checks are going to fail so this would be short-lived. If
> - * the allocation fails because of a memory shortage, then this is not a
> - * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
> - */
> if (!AllocSizeIsValid(newSize))
> return false;
>
> Wouldn't it be OK to drop entirely this check? I'm fine to keep it as
> a safety measure, but it should not be necessary now that it gets
> called only once the header is validated?
Yeah, now we're in "shouldn't happen" territory, and I'm not sure.
> Regarding the tests, I have been using this formula to produce the
> number of bytes until the next page boundary:
> select setting::int - ((pg_current_wal_lsn() - '0/0') % setting::int)
> from pg_settings where name = 'wal_block_size';
>
> Using pg_logical_emit_message() non-transactional with an empty set of
> strings generates records of 56 bytes, so I was thinking about the
> following:
> - Generate a bunch of small records with pg_logical_emit_message(), or
> records based on the threshold with the previous formula.
> - Loop until we reach a page limit, at 24 bytes (?).
> - Generate one last record to cut through.
> - Stop the node in immediate mode.
> - Write some garbage bytes on the last page generated to emulate the
> recycled contents and an allocation
> - Start the node, which should be able to startup.
> With wal_level = minimal, autovacuum = off and a large
> checkpoint_timeout, any other records are minimized. That's fancy,
> though.
>
> Do you think that this could work reliably?
Yeah, I think that sounds quite promising, and funnily enough I was
just now working on some Perl code that appends controlled junk to the
WAL in a test like that so we can try to hit all the error paths...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-15 03:00 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-15 03:00 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Tue, Aug 15, 2023 at 02:36:10PM +1200, Thomas Munro wrote:
> Yeah, I think that sounds quite promising, and funnily enough I was
> just now working on some Perl code that appends controlled junk to the
> WAL in a test like that so we can try to hit all the error paths...
I am pretty sure that adding some junk in a controlled manner is the
only cheap and rather-reliable way to get something..
Not sure if that will help, but what I was playing with some stuff in
the lines of:
-- Store the length up to page boundary.
select setting::int - ((pg_current_wal_insert_lsn() - '0/0') %
setting::int) as boundary from pg_settings where name = 'wal_block_size'
\gset
-- Generate record up to boundary (56 bytes for base size of the record,
-- stop at 12 bytes before the end of the page.
select pg_logical_emit_message(false, '', repeat('a', :boundary - 56 - 12));
Then by injecting some FF's on the last page written and forcing
replay I am able to force some of the error code paths, so I guess
that's what you were basically doing?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNrqTmKGnZwz9gPI@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-15 06:11 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-15 06:11 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Tue, Aug 15, 2023 at 12:00:30PM +0900, Michael Paquier wrote:
> Not sure if that will help, but what I was playing with some stuff in
> the lines of:
> -- Store the length up to page boundary.
> select setting::int - ((pg_current_wal_insert_lsn() - '0/0') %
> setting::int) as boundary from pg_settings where name = 'wal_block_size'
> \gset
> -- Generate record up to boundary (56 bytes for base size of the record,
> -- stop at 12 bytes before the end of the page.
> select pg_logical_emit_message(false, '', repeat('a', :boundary - 56 - 12));
>
> Then by injecting some FF's on the last page written and forcing
> replay I am able to force some of the error code paths, so I guess
> that's what you were basically doing?
I've been spending some extra time on this one and hacked a TAP test
that reliably reproduces the original issue, using a message similar
to what I mentioned in my previous messages. I guess that we could
use something like that:
2023-08-15 15:07:03.790 JST [8729] LOG: redo starts at 0/14EA428
2023-08-15 15:07:03.790 JST [8729] FATAL: invalid memory alloc
request size 4294969740 2023-08-15
15:07:03.791 JST [8726] LOG: startup process (PID 8729) exited with exit code 1
The proposed patches pass the test, HEAD does not. We may want to do
more with page boundaries, and more error patterns, but the idea looks
worth exploring more. At least this can be used to validate patches.
I've noticed while hacking the test that we don't do a XLogFlush()
after inserting the message's record, so we may lose it on crash.
That makes the test unstable except if an extra record is added after
the logical messages. The attached patch forces that for the sake of
the test, but I'm spawning a different thread as losing this data
looks like a bug to me.
--
Michael
Attachments:
[text/x-diff] 0001-Add-test-to-emulate-random-garbage-data-during-WAL-r.patch (4.1K, ../../ZNsXBFsFsKcCbP0q@paquier.xyz/2-0001-Add-test-to-emulate-random-garbage-data-during-WAL-r.patch)
download | inline diff:
From 958130099bc76534031acc4d3f2e94ccbcfa5451 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 15 Aug 2023 15:09:28 +0900
Subject: [PATCH] Add test to emulate random garbage data during WAL replay,
doing OOMs
This requires an extra flush wait at the end of LogLogicalMessage() to
make the test stable.
---
src/backend/replication/logical/message.c | 7 ++-
src/test/recovery/t/038_wal_invalid.pl | 69 +++++++++++++++++++++++
2 files changed, 75 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/038_wal_invalid.pl
diff --git a/src/backend/replication/logical/message.c b/src/backend/replication/logical/message.c
index c5de14afc6..cebe6c5d05 100644
--- a/src/backend/replication/logical/message.c
+++ b/src/backend/replication/logical/message.c
@@ -47,6 +47,7 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size,
bool transactional)
{
xl_logical_message xlrec;
+ XLogRecPtr lsn;
/*
* Force xid to be allocated if we're emitting a transactional message.
@@ -71,7 +72,11 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size,
/* allow origin filtering */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- return XLogInsert(RM_LOGICALMSG_ID, XLOG_LOGICAL_MESSAGE);
+ lsn = XLogInsert(RM_LOGICALMSG_ID, XLOG_LOGICAL_MESSAGE);
+
+ /* Make sure that the message hits disk before leaving */
+ XLogFlush(lsn);
+ return lsn;
}
/*
diff --git a/src/test/recovery/t/038_wal_invalid.pl b/src/test/recovery/t/038_wal_invalid.pl
new file mode 100644
index 0000000000..87a6e93734
--- /dev/null
+++ b/src/test/recovery/t/038_wal_invalid.pl
@@ -0,0 +1,69 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+# Test for code paths involving detection of incorrect WAL records.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Initialize and start node with wal_level = minimal and some
+# configuration aimed at minimizing the WAL activity of the cluster.
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->append_conf('postgresql.conf', q[
+wal_level = minimal
+autovacuum = off
+checkpoint_timeout = '30min'
+]);
+$node->start;
+
+# Generate two WAL records that will be used for the follow-up test,
+# using pg_logical_emit_message. This function returns the end LSN
+# of the record inserted, and the LSN of the first record is what
+# we use to invalidate the beginning of the second record. This
+# emulates some historical patterns where a record is read on a
+# page read from a recycled segment at the end of WAL. The second
+# record could be anything, but pg_logical_emit_message() does not
+# impact the cluster consistency.
+my $result = $node->safe_psql('postgres',
+ q{SELECT * FROM pg_walfile_name_offset(pg_logical_emit_message(false, 'a', 'a'))});
+my ($wal_segment_name, $lsn_offset) = split /\|/, $result;
+$lsn_offset = int($lsn_offset);
+
+$node->safe_psql('postgres',
+ q{SELECT pg_logical_emit_message(false, 'a', 'a')});
+
+# Shut down the cluster, to replay the previous records.
+$node->stop('immediate');
+
+# Add 4 bytes of garbage at the beginning of the second record, to
+# emulate the case of a recycled page. The previous offset points
+# at the beginning of the second record's xl_tot_len.
+my $pgdata = $node->data_dir;
+
+my $fh;
+open($fh, '+<', "$pgdata/pg_wal/$wal_segment_name")
+ or BAIL_OUT("open failed: $!");
+sysseek($fh, $lsn_offset, 0)
+ or BAIL_OUT("sysseek failed: $!");
+syswrite($fh, pack("L", 0xFFFFFFFF)) or BAIL_OUT("syswrite failed: $!");
+close($fh)
+ or BAIL_OUT("close failed: $!");
+
+# The node should be able to start.
+my $log_location = -s $node->logfile;
+$node->start;
+$result = $node->safe_psql('postgres', q(SELECT 1));
+is($result, "1", "node is up and running");
+
+# Confirm thatrecovery has finished with the expected error
+my $logfile = slurp_file($node->logfile, $log_location);
+ok( $logfile =~
+ qr/LOG: .* invalid magic number 0000 in WAL segment $wal_segment_name/,
+ "recovery completed up to the invalid record"
+);
+
+done_testing();
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZNsXBFsFsKcCbP0q@paquier.xyz/3-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-16 03:29 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-16 03:29 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Tue, Aug 15, 2023 at 6:11 PM Michael Paquier <michael@paquier.xyz> wrote:
> I've been spending some extra time on this one and hacked a TAP test
> that reliably reproduces the original issue, using a message similar
> to what I mentioned in my previous messages.
Nice.
I hacked on this idea for quite a long time yesterday and today and
came up with a set of tests for the main end-of-WAL conditions:
▶ 1/1 - xl_tot_len zero OK
▶ 1/1 - xl_tot_len short OK
▶ 1/1 - xl_prev bad OK
▶ 1/1 - xl_crc bad OK
▶ 1/1 - xlp_magic zero OK
▶ 1/1 - xlp_magic bad OK
▶ 1/1 - xlp_pageaddr bad OK
▶ 1/1 - xlp_info bad OK
▶ 1/1 - xlp_info lacks XLP_FIRST_IS_CONTRECORD OK
▶ 1/1 - xlp_rem_len bad OK
▶ 1/1 - xlp_magic zero (split record header) OK
▶ 1/1 - xlp_pageaddr bad (split record header) OK
▶ 1/1 - xlp_rem_len bad (split record header) OK
1/1 postgresql:recovery / recovery/038_end_of_wal OK
5.79s 13 subtests passed
It took me a while to come up with a workable way to get into the
record-header-splitting zone. Based on some of your clues about
flushing, I eventually realised I needed transactional messages, and I
built a kind of self-calibrating Rube Goldberg function around that.
It's terrible, and I'm sure we can do better.
I wonder what people think about putting internal details of the WAL
format into a Perl test like this. Obviously it requires maintenance,
since it knows the size and layout of a few things. I guess it'd be
allowed to fish a couple of those numbers out of the source.
Work in progress... I'm sure more useful checks could be added. One
thing that occurred to me while thinking about all this it that the
'treat malloc failure as end of WAL' thing you highlighted in another
thread is indeed completely bananas -- I didn't go digging, but
perhaps it was an earlier solution to the very same garbage xl_tot_len
problem, before 70b4f82a4b5 and now this xl_rem_len-based solution?
Attachments:
[application/x-patch] 0001-Add-TAP-tests-for-end-of-WAL-conditions.patch (12.5K, ../../CA+hUKGLhFdd-G1DCk9Ze3KnQ_2jUkxyXvGwV_Ha=DWMrDnGHng@mail.gmail.com/2-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch)
download | inline diff:
From fdabbaaca4f8f788b473ffeccc9e788b31ed5a87 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 15 Aug 2023 17:37:53 +1200
Subject: [PATCH] Add TAP tests for end-of-WAL conditions.
XXX
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..fefb5b6034
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,415 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test ways of detecting end of WAL.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Values that could be extracted from the source tree
+my $RECORD_HEADER_SIZE = 24;
+my $XLP_PAGE_MAGIC = 0xd113;
+my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
+ $node->data_dir,
+ $tli,
+ 0,
+ $segment);
+ return $wal_path;
+}
+
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+sub get_log_size
+{
+ my $node = shift;
+ return (stat $node->logfile)[7];
+}
+
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+}
+
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+sub record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # struct XLogRecord
+ return pack("IIQCCBBI",
+ $xl_tot_len,
+ $xl_xid,
+ $xl_prev,
+ $xl_info,
+ $xl_rmid,
+ 0,
+ 0,
+ $xl_crc);
+}
+
+sub page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # struct XLogPageHeaderData
+ return pack("SSIQI",
+ $xlp_magic,
+ $xlp_info,
+ $xlp_tli,
+ $xlp_pageaddr,
+ $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ {
+ $end_lsn = emit_message($node, 2000);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # calibrate our message size so that we can get closer 8 bytes at
+ # a time (XXX something better is surely possible)
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # adjust the message size until it causes 8 bytes changes in
+ # offset
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf('postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "create table t as select 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+
+
+my $end_lsn;
+my $prev_lsn;
+my $log_size;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
+ $log_size),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(23));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# need more pages, but xl_prev check fails first
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
+ $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("incorrect resource manager data checksum in record at .*",
+ $log_size),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# good xl_prev, we hit zero page next (zero magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+#my $prev_lsn = emit_message($node, 0);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# good xl_prev, we hit garbage page next (bad magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header(0xcafe,
+ 0,
+ 1,
+ 0));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0x1234,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# and we'll also check xlp_pageaddr before any header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# we'll also discover that xlp_rem_len doesn't add up before any
+# header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.41.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-16 06:17 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-16 06:17 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Wed, Aug 16, 2023 at 03:29:49PM +1200, Thomas Munro wrote:
> I hacked on this idea for quite a long time yesterday and today and
> came up with a set of tests for the main end-of-WAL conditions:
>
> ▶ 1/1 - xl_tot_len zero OK
> ▶ 1/1 - xl_tot_len short OK
> ▶ 1/1 - xl_prev bad OK
> ▶ 1/1 - xl_crc bad OK
> ▶ 1/1 - xlp_magic zero OK
> ▶ 1/1 - xlp_magic bad OK
> ▶ 1/1 - xlp_pageaddr bad OK
> ▶ 1/1 - xlp_info bad OK
> ▶ 1/1 - xlp_info lacks XLP_FIRST_IS_CONTRECORD OK
> ▶ 1/1 - xlp_rem_len bad OK
> ▶ 1/1 - xlp_magic zero (split record header) OK
> ▶ 1/1 - xlp_pageaddr bad (split record header) OK
> ▶ 1/1 - xlp_rem_len bad (split record header) OK
> 1/1 postgresql:recovery / recovery/038_end_of_wal OK
> 5.79s 13 subtests passed
Nice.
> It took me a while to come up with a workable way to get into the
> record-header-splitting zone. Based on some of your clues about
> flushing, I eventually realised I needed transactional messages, and I
> built a kind of self-calibrating Rube Goldberg function around that.
> It's terrible, and I'm sure we can do better.
So that's advance_to_record_splitting_zone(), with the idea to loop
across multiple records. The use of transactional messages ensures
the flush and the stability of the test when the server is stopped,
but it makes the records a bit longer than the non-transactional
messages.
> I wonder what people think about putting internal details of the WAL
> format into a Perl test like this. Obviously it requires maintenance,
> since it knows the size and layout of a few things. I guess it'd be
> allowed to fish a couple of those numbers out of the source.
The format of the WAL header does not change a lot across the years,
so I'd be OK with that. If that's an issue, the scope of the test
could always be restricted a bit.
> Work in progress... I'm sure more useful checks could be added. One
> thing that occurred to me while thinking about all this it that the
> 'treat malloc failure as end of WAL' thing you highlighted in another
> thread is indeed completely bananas -- I didn't go digging, but
> perhaps it was an earlier solution to the very same garbage xl_tot_len
> problem, before 70b4f82a4b5 and now this xl_rem_len-based solution?
It is bananas. Still, the problem of the other thread is similar to
what's here, and different at the same time because it can happen with
a valid record if the host is under memory pressure, and we should
still allow a standby to loop and continue in this case. Perhaps also
a startup process doing only crash recovery. This thread's problem is
dependent on this one, which is why this one needs to happen first: we
need to clarify the boundary between a real OOM and garbage indicating
the end of WAL.
+my $RECORD_HEADER_SIZE = 24;
+my $XLP_PAGE_MAGIC = 0xd113;
+my $XLP_FIRST_IS_CONTRECORD = 0x1;
This could use something like check_pg_config() to find the header
contents for the last two ones, but there's no easy way to extract the
size of the header struct, is there? Or perhaps perl has a tool for
that.. Anything I can read on the matter tells to use pack() to make
the size estimations predictible.
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ {
+ $end_lsn = emit_message($node, 2000)
The hardcoded 2000 feels magic here, but I get that you just want to
get away from the page border, as well. I'd be OK with more comments.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
Now the backend also has pg_walfile_name_offset(). I tend to rely on
the backend logic as much as I can for tests.
+sub start_of_page
Not sure this one is needed, only being called in
start_of_next_page().
+sub get_insert_lsn
Name is confusing with Cluster::lsn('insert') available, perhaps this
should be named get_insert_lsn_in_bytes.
As a whole, I find that pretty cool. Not sure if I would backpatch
the test, so my opinion would be to use that on HEAD and let it
stabilize there.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZNxqFgo243DOar4L@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-18 00:20 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-18 00:20 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Wed, Aug 16, 2023 at 03:17:58PM +0900, Michael Paquier wrote:
> As a whole, I find that pretty cool. Not sure if I would backpatch
> the test, so my opinion would be to use that on HEAD and let it
> stabilize there.
Thomas, is there anything I can do to help with this patch and make it
move on?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZN65RHuB+qNS%2FG9Q@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-18 02:30 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-08-18 02:30 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Fri, Aug 18, 2023 at 12:20 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Aug 16, 2023 at 03:17:58PM +0900, Michael Paquier wrote:
> > As a whole, I find that pretty cool. Not sure if I would backpatch
> > the test, so my opinion would be to use that on HEAD and let it
> > stabilize there.
>
> Thomas, is there anything I can do to help with this patch and make it
> move on?
Give me a couple of days and I'll look into how back-patchable the
tests can be made, and see what else we can test. Perhaps it's not
strictly necessary to back-patch the fix further than 15, but I think
we should definitely consider it, and I don't like the idea of not
having the tests accompanying the change.
If you have any ideas about how to write a more efficient version of
advance_to_record_splitting_zone() (or I guess that should really be
advance_to_record_header_splitting_zone()), and generally how to make
the perl better, and how to get those constants we need from the
source or binaries, then I'm all ears.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-18 04:55 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-18 04:55 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Fri, Aug 18, 2023 at 02:30:31PM +1200, Thomas Munro wrote:
> Give me a couple of days and I'll look into how back-patchable the
> tests can be made, and see what else we can test. Perhaps it's not
> strictly necessary to back-patch the fix further than 15, but I think
> we should definitely consider it, and I don't like the idea of not
> having the tests accompanying the change.
Okay, cool.
> If you have any ideas about how to write a more efficient version of
> advance_to_record_splitting_zone() (or I guess that should really be
> advance_to_record_header_splitting_zone()), and generally how to make
> the perl better,
Yeah, I think that there are ways to reduce the number of records
generated.
> and how to get those constants we need from the
> source or binaries, then I'm all ears.
A new perl routine able to do a pg_control --includedir that scans a
defined header with a regexp would be able to make the job for the
constants. If you're interested, I can code that up.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZN75xA4v14ImLvhy@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-08-20 23:32 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-08-20 23:32 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Fri, Aug 18, 2023 at 01:55:32PM +0900, Michael Paquier wrote:
> A new perl routine able to do a pg_control --includedir that scans a
> defined header with a regexp would be able to make the job for the
> constants. If you're interested, I can code that up.
Please find attached a patch, that applies on top of the TAP test,
to implement a logic able to extract from header files the values
wanted. This is Utils::scan_header(), for example:
my @scan_result = scan_header('server/access/xlog_internal.h',
'#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
my $XLP_PAGE_MAGIC = hex($scan_result[0]);
This uses group capture so as it is possible to get more than one
field. Guessing the size of the C structure is possible, but I've
found these options to be a bit awkward implementation-wise:
- Config would not help for the Postgres-specific declarations, like
TimeLineId, etc.
- I saw something that was specific to GCC.. cannot put my finger on
it now.
Another possibility would be to store the size in a different variable
in xlog_internal.h, coupled with a StaticAssertDecl() to make sure
that it matches with the real size. While it has the same maintenance
cost as RECORD_HEADER_SIZE in the test itself, that's catching a
mismatching size at compilation-time rather than at test-time.
I am not sure that I'll be able to do more on this topic this week, at
least that's some progress.
--
Michael
Attachments:
[text/x-diff] v2-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch (12.8K, ../../ZOKil9ifJmmAb6di@paquier.xyz/2-v2-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch)
download | inline diff:
From b71a2aab7c56e99a4d3b6c713264b3c20a1a3499 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 15 Aug 2023 17:37:53 +1200
Subject: [PATCH v2 1/3] Add TAP tests for end-of-WAL conditions.
XXX
---
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 415 ++++++++++++++++++++++++++
2 files changed, 416 insertions(+)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..fefb5b6034
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,415 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test ways of detecting end of WAL.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Values that could be extracted from the source tree
+my $RECORD_HEADER_SIZE = 24;
+my $XLP_PAGE_MAGIC = 0xd113;
+my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
+ $node->data_dir,
+ $tli,
+ 0,
+ $segment);
+ return $wal_path;
+}
+
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+sub get_log_size
+{
+ my $node = shift;
+ return (stat $node->logfile)[7];
+}
+
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+}
+
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+sub record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # struct XLogRecord
+ return pack("IIQCCBBI",
+ $xl_tot_len,
+ $xl_xid,
+ $xl_prev,
+ $xl_info,
+ $xl_rmid,
+ 0,
+ 0,
+ $xl_crc);
+}
+
+sub page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # struct XLogPageHeaderData
+ return pack("SSIQI",
+ $xlp_magic,
+ $xlp_info,
+ $xlp_tli,
+ $xlp_pageaddr,
+ $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ {
+ $end_lsn = emit_message($node, 2000);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # calibrate our message size so that we can get closer 8 bytes at
+ # a time (XXX something better is surely possible)
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # adjust the message size until it causes 8 bytes changes in
+ # offset
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf('postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "create table t as select 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+
+
+my $end_lsn;
+my $prev_lsn;
+my $log_size;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
+ $log_size),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(23));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# need more pages, but xl_prev check fails first
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
+ $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("incorrect resource manager data checksum in record at .*",
+ $log_size),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# good xl_prev, we hit zero page next (zero magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+#my $prev_lsn = emit_message($node, 0);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# good xl_prev, we hit garbage page next (bad magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header(0xcafe,
+ 0,
+ 1,
+ 0));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0x1234,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# and we'll also check xlp_pageaddr before any header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# we'll also discover that xlp_rem_len doesn't add up before any
+# header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/x-diff] v2-0002-Don-t-trust-unvalidated-xl_tot_len.patch (7.8K, ../../ZOKil9ifJmmAb6di@paquier.xyz/3-v2-0002-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From 7b3dc9aa37c571091dcc73e3e30567803bffb5f1 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v2 2/3] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally in old and
new branches using pre-existing page-level validation.
Whenever a record header spans a page boundary, we just need to defer
all memory allocation based on xl_tot_len until we've read the following
page. At that point we know that its first sector is not recycled data,
by checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 12. This failure mode is rare, and 11 is about to go out
of support, so we decided not to change its final release.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 110 +++++++++++++-----------
1 file changed, 61 insertions(+), 49 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..d5ed139131 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -203,18 +203,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
@@ -669,41 +657,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +691,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +792,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +862,25 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /* We failed to allocate memory for an oversized record. */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +909,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
--
2.40.1
[text/x-diff] v2-0003-Implement-routine-to-scan-and-extra-values-from-h.patch (2.7K, ../../ZOKil9ifJmmAb6di@paquier.xyz/4-v2-0003-Implement-routine-to-scan-and-extra-values-from-h.patch)
download | inline diff:
From 87912b0d0dc272b20ec9659479eebdfc73d78c79 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 21 Aug 2023 08:17:32 +0900
Subject: [PATCH v2 3/3] Implement routine to scan and extra values from
headers
---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 ++++++++++++++++++++++++++
src/test/recovery/t/038_end_of_wal.pl | 12 +++++--
2 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..80ea9b0759 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
index fefb5b6034..578026e692 100644
--- a/src/test/recovery/t/038_end_of_wal.pl
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -11,10 +11,16 @@ use Fcntl qw(SEEK_SET);
use integer; # causes / operator to use integer math
-# Values that could be extracted from the source tree
+# Header size of record header.
my $RECORD_HEADER_SIZE = 24;
-my $XLP_PAGE_MAGIC = 0xd113;
-my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Fields retrieved from headers.
+my @scan_result = scan_header('server/access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_header('server/access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
# Values queried from the server
my $WAL_SEGMENT_SIZE;
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZOKil9ifJmmAb6di@paquier.xyz/5-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-02 01:29 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-02 01:29 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Aug 21, 2023 at 08:32:39AM +0900, Michael Paquier wrote:
> I am not sure that I'll be able to do more on this topic this week, at
> least that's some progress.
Some time later..
I have spent more time and thoughts on the whole test suite, finishing
with the attached 0003 that applies on top of your own patches. I am
really looking forward to making this whole logic more robust, so as
WAL replay can be made itself more robust for the OOM/end-of-wal
detection on HEAD for standbys and crash recovery.
While looking at the whole, I have considered a few things that may
make the test cleaner, like:
- Calculating the segment name and its offset from the end_lsn of a
record directly from the backend, but it felt inelegant to pass
through more subroutine layers the couple ($segment, offset) rather
than just a LSN, so guessing the segment number and the offset while
the cluster is offline if OK by me.
- The TLI can be queried from the server rather than hardcoded.
- I've been thinking about bundling the tests of each sub-section in
their own subroutine, but that felt a bit awkward, particularly for
the part where we need a correct $prev_lsn in the record header
written to enforce other code paths.
- The test needs better documentation. One of the things I kept
staring at is cross-checking pack() and the dependency to the C
structures, so I have added more details there, explaining more the
whys and the hows.
I have also looked again at the C code for a few hours, and still got
the impression that this is rather solid. There are two things that
may be better:
- Document at the top of allocate_recordbuf() that this should never
be called with a length coming from a header until it is validated.
- Removing AllocSizeIsValid() for the non-FRONTEND path should be OK.
What do you think?
--
Michael
Attachments:
[text/x-diff] v3-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch (12.8K, ../../ZPKQA+HHqLVRFBSs@paquier.xyz/2-v3-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch)
download | inline diff:
From 8a8dd8252dd3b096d1c652ab78671d6457ecc95a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 15 Aug 2023 17:37:53 +1200
Subject: [PATCH v3 1/3] Add TAP tests for end-of-WAL conditions.
XXX
---
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 415 ++++++++++++++++++++++++++
2 files changed, 416 insertions(+)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..fefb5b6034
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,415 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test ways of detecting end of WAL.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Values that could be extracted from the source tree
+my $RECORD_HEADER_SIZE = 24;
+my $XLP_PAGE_MAGIC = 0xd113;
+my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
+ $node->data_dir,
+ $tli,
+ 0,
+ $segment);
+ return $wal_path;
+}
+
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+sub get_log_size
+{
+ my $node = shift;
+ return (stat $node->logfile)[7];
+}
+
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+}
+
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+sub record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # struct XLogRecord
+ return pack("IIQCCBBI",
+ $xl_tot_len,
+ $xl_xid,
+ $xl_prev,
+ $xl_info,
+ $xl_rmid,
+ 0,
+ 0,
+ $xl_crc);
+}
+
+sub page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # struct XLogPageHeaderData
+ return pack("SSIQI",
+ $xlp_magic,
+ $xlp_info,
+ $xlp_tli,
+ $xlp_pageaddr,
+ $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ {
+ $end_lsn = emit_message($node, 2000);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # calibrate our message size so that we can get closer 8 bytes at
+ # a time (XXX something better is surely possible)
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # adjust the message size until it causes 8 bytes changes in
+ # offset
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf('postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "create table t as select 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+
+
+my $end_lsn;
+my $prev_lsn;
+my $log_size;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
+ $log_size),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(23));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# need more pages, but xl_prev check fails first
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
+ $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("incorrect resource manager data checksum in record at .*",
+ $log_size),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# good xl_prev, we hit zero page next (zero magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+#my $prev_lsn = emit_message($node, 0);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# good xl_prev, we hit garbage page next (bad magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header(0xcafe,
+ 0,
+ 1,
+ 0));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0x1234,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# and we'll also check xlp_pageaddr before any header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# we'll also discover that xlp_rem_len doesn't add up before any
+# header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/x-diff] v3-0002-Don-t-trust-unvalidated-xl_tot_len.patch (7.8K, ../../ZPKQA+HHqLVRFBSs@paquier.xyz/3-v3-0002-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From 4db94cbdf5075b84a2c8ccc257d90d3671624413 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v3 2/3] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally in old and
new branches using pre-existing page-level validation.
Whenever a record header spans a page boundary, we just need to defer
all memory allocation based on xl_tot_len until we've read the following
page. At that point we know that its first sector is not recycled data,
by checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 12. This failure mode is rare, and 11 is about to go out
of support, so we decided not to change its final release.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 110 +++++++++++++-----------
1 file changed, 61 insertions(+), 49 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..d5ed139131 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -203,18 +203,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
@@ -669,41 +657,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +691,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +792,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +862,25 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /* We failed to allocate memory for an oversized record. */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +909,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
--
2.40.1
[text/x-diff] v3-0003-Adjust-few-things-on-top-of-Thomas-patch-series.patch (22.5K, ../../ZPKQA+HHqLVRFBSs@paquier.xyz/4-v3-0003-Adjust-few-things-on-top-of-Thomas-patch-series.patch)
download | inline diff:
From 6e0afd1f212714641b764ecf7ecc9ce1ee83c414 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Sat, 2 Sep 2023 10:18:39 +0900
Subject: [PATCH v3 3/3] Adjust few things on top of Thomas' patch series
---
src/backend/access/transam/xlogreader.c | 10 +-
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 386 +++++++++++++-----------
3 files changed, 253 insertions(+), 186 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d5ed139131..5bc7639a00 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,13 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..d7d59eb745 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
index fefb5b6034..321f5965d1 100644
--- a/src/test/recovery/t/038_end_of_wal.pl
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -1,6 +1,7 @@
# Copyright (c) 2023, PostgreSQL Global Development Group
#
-# Test ways of detecting end of WAL.
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
use strict;
use warnings;
@@ -9,42 +10,44 @@ use PostgreSQL::Test::Utils;
use Test::More;
use Fcntl qw(SEEK_SET);
-use integer; # causes / operator to use integer math
+use integer; # causes / operator to use integer math
-# Values that could be extracted from the source tree
+# Header size of record header.
my $RECORD_HEADER_SIZE = 24;
-my $XLP_PAGE_MAGIC = 0xd113;
-my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_header('server/access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_header('server/access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
# Values queried from the server
my $WAL_SEGMENT_SIZE;
my $WAL_BLOCK_SIZE;
+my $TLI;
+# Build path of a WAL segment.
sub wal_segment_path
{
my $node = shift;
my $tli = shift;
my $segment = shift;
- my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
- $node->data_dir,
- $tli,
- 0,
- $segment);
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
return $wal_path;
}
+# Calculate from a LSN (in bytes) its segment number and its offset.
sub lsn_to_segment_and_offset
{
my $lsn = shift;
return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
}
-sub get_log_size
-{
- my $node = shift;
- return (stat $node->logfile)[7];
-}
-
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
sub write_wal
{
my $node = shift;
@@ -61,30 +64,37 @@ sub write_wal
close $fh;
}
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
sub emit_message
{
my $node = shift;
my $size = shift;
- return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
}
+# Get the current insert LSN of a node, in bytes.
sub get_insert_lsn
{
my $node = shift;
- return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
}
+# Get GUC value, converted to an int.
sub get_int_setting
{
my $node = shift;
my $name = shift;
- return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
-}
-
-sub format_lsn
-{
- my $lsn = shift;
- return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
}
sub start_of_page
@@ -99,7 +109,10 @@ sub start_of_next_page
return start_of_page($lsn) + $WAL_BLOCK_SIZE;
}
-sub record_header
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
{
my $xl_tot_len = shift;
my $xl_xid = shift || 0;
@@ -108,19 +121,22 @@ sub record_header
my $xl_rmid = shift || 0;
my $xl_crc = shift || 0;
- # struct XLogRecord
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
return pack("IIQCCBBI",
- $xl_tot_len,
- $xl_xid,
- $xl_prev,
- $xl_info,
- $xl_rmid,
- 0,
- 0,
- $xl_crc);
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
}
-sub page_header
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
{
my $xlp_magic = shift;
my $xlp_info = shift || 0;
@@ -128,32 +144,37 @@ sub page_header
my $xlp_pageaddr = shift || 0;
my $xlp_rem_len = shift || 0;
- # struct XLogPageHeaderData
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
return pack("SSIQI",
- $xlp_magic,
- $xlp_info,
- $xlp_tli,
- $xlp_pageaddr,
- $xlp_rem_len);
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
}
# Make sure we are far away enough from the end of a page that we could insert
-# a couple of small records.
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
sub advance_out_of_record_splitting_zone
{
my $node = shift;
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
my $end_lsn = get_insert_lsn($node);
my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
{
- $end_lsn = emit_message($node, 2000);
+ $end_lsn = emit_message($node, $page_threshold);
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
}
return $end_lsn;
}
-# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
sub advance_to_record_splitting_zone
{
my $node = shift;
@@ -161,15 +182,15 @@ sub advance_to_record_splitting_zone
my $end_lsn = get_insert_lsn($node);
my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- # get fairly close to the end of a page in big steps
+ # Get fairly close to the end of a page in big steps
while ($page_offset <= $WAL_BLOCK_SIZE - 512)
{
$end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
}
- # calibrate our message size so that we can get closer 8 bytes at
- # a time (XXX something better is surely possible)
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
my $message_size = $WAL_BLOCK_SIZE - 80;
while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
{
@@ -178,8 +199,8 @@ sub advance_to_record_splitting_zone
my $old_offset = $page_offset;
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- # adjust the message size until it causes 8 bytes changes in
- # offset
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
my $delta = $page_offset - $old_offset;
if ($delta > 8)
{
@@ -193,223 +214,230 @@ sub advance_to_record_splitting_zone
return $end_lsn;
}
-
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
my $node = PostgreSQL::Test::Cluster->new("node");
$node->init;
-$node->append_conf('postgresql.conf',
- q[wal_level = minimal
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
autovacuum = off
checkpoint_timeout = '30min'
]);
$node->start;
-$node->safe_psql('postgres', "create table t as select 42");
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
-
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
my $end_lsn;
my $prev_lsn;
-my $log_size;
###########################################################################
note "Single-page end-of-WAL detection";
###########################################################################
-# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-$log_size = get_log_size($node);
+my $log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
- $log_size),
- "xl_tot_len zero");
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
-# xl_tot_len is < 24 (presumably recycled garbage)
+# xl_tot_len is < 24 (presumably recycled garbage).
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(23));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
- $log_size),
- "xl_tot_len short");
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
-# need more pages, but xl_prev check fails first
+# Need more pages, but xl_prev check fails first.
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
- $log_size),
- "xl_prev bad");
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
-# xl_crc check fails
+# xl_crc check fails.
advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 10);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
-$log_size = get_log_size($node);
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("incorrect resource manager data checksum in record at .*",
- $log_size),
- "xl_crc bad");
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
###########################################################################
note "Multi-page end-of-WAL detection, header is not split";
###########################################################################
-# good xl_prev, we hit zero page next (zero magic)
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
$prev_lsn = advance_out_of_record_splitting_zone($node);
-#my $prev_lsn = emit_message($node, 0);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero");
+ "xlp_magic zero");
-# good xl_prev, we hit garbage page next (bad magic)
+# Good xl_prev, we hit garbage page next (bad magic).
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header(0xcafe,
- 0,
- 1,
- 0));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
- "xlp_magic bad");
+ "xlp_magic bad");
-# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0,
- 1,
- 0xbaaaaaad));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad");
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
-# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0x1234,
- 1,
- start_of_next_page($end_lsn)));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
- "xlp_info bad");
+ "xlp_info bad");
-# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0,
- 1,
- start_of_next_page($end_lsn)));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("there is no contrecord flag at .*", $log_size),
- "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
-# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- start_of_next_page($end_lsn),
- 123456));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad");
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
###########################################################################
note "Multi-page, but header is split, so page checks are done first";
###########################################################################
-# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero (split record header)");
+ "xlp_magic zero (split record header)");
-# and we'll also check xlp_pageaddr before any header checks
+# And we'll also check xlp_pageaddr before any header checks.
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- 0xbaaaaaad));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad (split record header)");
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
-# we'll also discover that xlp_rem_len doesn't add up before any
-# header checks
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- start_of_next_page($end_lsn),
- 123456));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad (split record header)");
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
done_testing();
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZPKQA+HHqLVRFBSs@paquier.xyz/5-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-02 11:51 Sergei Kornilov <sk@zsrv.org>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Sergei Kornilov @ 2023-09-02 11:51 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
Hello
I have an error from the new test due to Test/Utils.pm new scan_header function.
I applied all 3 patches and use
./configure --prefix=/home/melkij/tmp/pgdev/inst --enable-cassert --enable-debug CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer" --enable-tap-tests
make -s world
make check-world
Then I see:
Test Summary Report
-------------------
t/038_end_of_wal.pl (Wstat: 512 (exited 2) Tests: 0 Failed: 0)
Non-zero exit status: 2
Parse errors: No plan found in TAP output
Corresponding tmp_check/log/regress_log_038_end_of_wal show error:
No such file or directory at /home/melkij/tmp/srcs/postgresql/src/test/recovery/../../../src/test/perl/PostgreSQL/Test/Utils.pm line 725.
pg_config from scan_header function returns /home/melkij/tmp/srcs/postgresql/tmp_install/home/melkij/tmp/pgdev/inst/include , then the function tries to concatenate server/access/xlog_internal.h, while the required file is in postgresql/server/access/xlog_internal.h from this directory.
If I change the test like this for both scan_header, then it succeeds.
- my @scan_result = scan_header('server/access/xlog_internal.h',
+ my @scan_result = scan_header('postgresql/server/access/xlog_internal.h',
Is there something wrong with my setup?
regards, Sergei
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-02 23:42 Michael Paquier <michael@paquier.xyz>
parent: Sergei Kornilov <sk@zsrv.org>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-02 23:42 UTC (permalink / raw)
To: Sergei Kornilov <sk@zsrv.org>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
On Sat, Sep 02, 2023 at 02:51:48PM +0300, Sergei Kornilov wrote:
> Corresponding tmp_check/log/regress_log_038_end_of_wal show error:
>
> No such file or directory at /home/melkij/tmp/srcs/postgresql/src/test/recovery/../../../src/test/perl/PostgreSQL/Test/Utils.pm line 725.
>
> pg_config from scan_header function returns
> /home/melkij/tmp/srcs/postgresql/tmp_install/home/melkij/tmp/pgdev/inst/include
> , then the function tries to concatenate
> server/access/xlog_internal.h, while the required file is in
> postgresql/server/access/xlog_internal.h from this directory.
>
> If I change the test like this for both scan_header, then it succeeds.
>
> - my @scan_result = scan_header('server/access/xlog_internal.h',
> + my @scan_result = scan_header('postgresql/server/access/xlog_internal.h',
>
> Is there something wrong with my setup?
I think that's just a mistake I did, as there should not even be a
need for having to specify "server/" for the header path. If you
switch scan_header() in Utils.pm to use --includedir-server rather
than --includedir, does it work correctly for you?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZPPIf3Uzp9iFrE6M@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-03 05:43 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-03 05:43 UTC (permalink / raw)
To: Sergei Kornilov <sk@zsrv.org>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
On Sun, Sep 03, 2023 at 08:42:55AM +0900, Michael Paquier wrote:
> I think that's just a mistake I did, as there should not even be a
> need for having to specify "server/" for the header path. If you
> switch scan_header() in Utils.pm to use --includedir-server rather
> than --includedir, does it work correctly for you?
In short, I mean something like the v4 attached. Feel free to use it
for your tests.
--
Michael
Attachments:
[text/x-diff] v4-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch (12.8K, ../../ZPQdHY4WDR+2ejwq@paquier.xyz/2-v4-0001-Add-TAP-tests-for-end-of-WAL-conditions.patch)
download | inline diff:
From 8a8dd8252dd3b096d1c652ab78671d6457ecc95a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 15 Aug 2023 17:37:53 +1200
Subject: [PATCH v4 1/3] Add TAP tests for end-of-WAL conditions.
XXX
---
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 415 ++++++++++++++++++++++++++
2 files changed, 416 insertions(+)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..fefb5b6034
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,415 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test ways of detecting end of WAL.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Values that could be extracted from the source tree
+my $RECORD_HEADER_SIZE = 24;
+my $XLP_PAGE_MAGIC = 0xd113;
+my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
+ $node->data_dir,
+ $tli,
+ 0,
+ $segment);
+ return $wal_path;
+}
+
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+sub get_log_size
+{
+ my $node = shift;
+ return (stat $node->logfile)[7];
+}
+
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+}
+
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+sub record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # struct XLogRecord
+ return pack("IIQCCBBI",
+ $xl_tot_len,
+ $xl_xid,
+ $xl_prev,
+ $xl_info,
+ $xl_rmid,
+ 0,
+ 0,
+ $xl_crc);
+}
+
+sub page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # struct XLogPageHeaderData
+ return pack("SSIQI",
+ $xlp_magic,
+ $xlp_info,
+ $xlp_tli,
+ $xlp_pageaddr,
+ $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ {
+ $end_lsn = emit_message($node, 2000);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # calibrate our message size so that we can get closer 8 bytes at
+ # a time (XXX something better is surely possible)
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # adjust the message size until it causes 8 bytes changes in
+ # offset
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf('postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "create table t as select 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+
+
+my $end_lsn;
+my $prev_lsn;
+my $log_size;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
+ $log_size),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage)
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(23));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# need more pages, but xl_prev check fails first
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
+ $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("incorrect resource manager data checksum in record at .*",
+ $log_size),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# good xl_prev, we hit zero page next (zero magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+#my $prev_lsn = emit_message($node, 0);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# good xl_prev, we hit garbage page next (bad magic)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header(0xcafe,
+ 0,
+ 1,
+ 0));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0x1234,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ 0,
+ 1,
+ start_of_next_page($end_lsn)));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 42,
+ $prev_lsn));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# and we'll also check xlp_pageaddr before any header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ 0xbaaaaaad));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# we'll also discover that xlp_rem_len doesn't add up before any
+# header checks
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
+ 0,
+ 0xdeadbeef));
+write_wal($node, 1, start_of_next_page($end_lsn),
+ page_header($XLP_PAGE_MAGIC,
+ $XLP_FIRST_IS_CONTRECORD,
+ 1,
+ start_of_next_page($end_lsn),
+ 123456));
+$log_size = get_log_size($node);
+$node->start;
+ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/x-diff] v4-0002-Don-t-trust-unvalidated-xl_tot_len.patch (7.8K, ../../ZPQdHY4WDR+2ejwq@paquier.xyz/3-v4-0002-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From 4db94cbdf5075b84a2c8ccc257d90d3671624413 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v4 2/3] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally in old and
new branches using pre-existing page-level validation.
Whenever a record header spans a page boundary, we just need to defer
all memory allocation based on xl_tot_len until we've read the following
page. At that point we know that its first sector is not recycled data,
by checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 12. This failure mode is rare, and 11 is about to go out
of support, so we decided not to change its final release.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 110 +++++++++++++-----------
1 file changed, 61 insertions(+), 49 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..d5ed139131 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -203,18 +203,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
#ifndef FRONTEND
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
if (!AllocSizeIsValid(newSize))
return false;
@@ -669,41 +657,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +691,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +792,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +862,25 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /* We failed to allocate memory for an oversized record. */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +909,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
--
2.40.1
[text/x-diff] v4-0003-Adjust-few-things-on-top-of-Thomas-patch-series.patch (22.6K, ../../ZPQdHY4WDR+2ejwq@paquier.xyz/4-v4-0003-Adjust-few-things-on-top-of-Thomas-patch-series.patch)
download | inline diff:
From 2a7d5d6a660bdbb04b50515e9b09bee4e8f694d8 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Sun, 3 Sep 2023 14:38:27 +0900
Subject: [PATCH v4 3/3] Adjust few things on top of Thomas' patch series
---
src/backend/access/transam/xlogreader.c | 10 +-
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 386 +++++++++++++-----------
3 files changed, 253 insertions(+), 186 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d5ed139131..5bc7639a00 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,13 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..9de34a2260 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
index fefb5b6034..aa5eec7c03 100644
--- a/src/test/recovery/t/038_end_of_wal.pl
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -1,6 +1,7 @@
# Copyright (c) 2023, PostgreSQL Global Development Group
#
-# Test ways of detecting end of WAL.
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
use strict;
use warnings;
@@ -9,42 +10,44 @@ use PostgreSQL::Test::Utils;
use Test::More;
use Fcntl qw(SEEK_SET);
-use integer; # causes / operator to use integer math
+use integer; # causes / operator to use integer math
-# Values that could be extracted from the source tree
+# Header size of record header.
my $RECORD_HEADER_SIZE = 24;
-my $XLP_PAGE_MAGIC = 0xd113;
-my $XLP_FIRST_IS_CONTRECORD = 0x1;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
# Values queried from the server
my $WAL_SEGMENT_SIZE;
my $WAL_BLOCK_SIZE;
+my $TLI;
+# Build path of a WAL segment.
sub wal_segment_path
{
my $node = shift;
my $tli = shift;
my $segment = shift;
- my $wal_path = sprintf("%s/pg_wal/%08X%08X%08X",
- $node->data_dir,
- $tli,
- 0,
- $segment);
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
return $wal_path;
}
+# Calculate from a LSN (in bytes) its segment number and its offset.
sub lsn_to_segment_and_offset
{
my $lsn = shift;
return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
}
-sub get_log_size
-{
- my $node = shift;
- return (stat $node->logfile)[7];
-}
-
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
sub write_wal
{
my $node = shift;
@@ -61,30 +64,37 @@ sub write_wal
close $fh;
}
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
sub emit_message
{
my $node = shift;
my $size = shift;
- return int($node->safe_psql('postgres', "select pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"));
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
}
+# Get the current insert LSN of a node, in bytes.
sub get_insert_lsn
{
my $node = shift;
- return int($node->safe_psql('postgres', "select pg_current_wal_insert_lsn() - '0/0'"));
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
}
+# Get GUC value, converted to an int.
sub get_int_setting
{
my $node = shift;
my $name = shift;
- return int($node->safe_psql('postgres', "select setting from pg_settings where name = '$name'"));
-}
-
-sub format_lsn
-{
- my $lsn = shift;
- return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
}
sub start_of_page
@@ -99,7 +109,10 @@ sub start_of_next_page
return start_of_page($lsn) + $WAL_BLOCK_SIZE;
}
-sub record_header
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
{
my $xl_tot_len = shift;
my $xl_xid = shift || 0;
@@ -108,19 +121,22 @@ sub record_header
my $xl_rmid = shift || 0;
my $xl_crc = shift || 0;
- # struct XLogRecord
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
return pack("IIQCCBBI",
- $xl_tot_len,
- $xl_xid,
- $xl_prev,
- $xl_info,
- $xl_rmid,
- 0,
- 0,
- $xl_crc);
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
}
-sub page_header
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
{
my $xlp_magic = shift;
my $xlp_info = shift || 0;
@@ -128,32 +144,37 @@ sub page_header
my $xlp_pageaddr = shift || 0;
my $xlp_rem_len = shift || 0;
- # struct XLogPageHeaderData
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
return pack("SSIQI",
- $xlp_magic,
- $xlp_info,
- $xlp_tli,
- $xlp_pageaddr,
- $xlp_rem_len);
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
}
# Make sure we are far away enough from the end of a page that we could insert
-# a couple of small records.
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
sub advance_out_of_record_splitting_zone
{
my $node = shift;
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
my $end_lsn = get_insert_lsn($node);
my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- while ($page_offset >= $WAL_BLOCK_SIZE - 2000)
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
{
- $end_lsn = emit_message($node, 2000);
+ $end_lsn = emit_message($node, $page_threshold);
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
}
return $end_lsn;
}
-# Advance so close to the end of a page that an XLogRecordHeader wouldn't fit.
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
sub advance_to_record_splitting_zone
{
my $node = shift;
@@ -161,15 +182,15 @@ sub advance_to_record_splitting_zone
my $end_lsn = get_insert_lsn($node);
my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- # get fairly close to the end of a page in big steps
+ # Get fairly close to the end of a page in big steps
while ($page_offset <= $WAL_BLOCK_SIZE - 512)
{
$end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
}
- # calibrate our message size so that we can get closer 8 bytes at
- # a time (XXX something better is surely possible)
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
my $message_size = $WAL_BLOCK_SIZE - 80;
while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
{
@@ -178,8 +199,8 @@ sub advance_to_record_splitting_zone
my $old_offset = $page_offset;
$page_offset = $end_lsn % $WAL_BLOCK_SIZE;
- # adjust the message size until it causes 8 bytes changes in
- # offset
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
my $delta = $page_offset - $old_offset;
if ($delta > 8)
{
@@ -193,223 +214,230 @@ sub advance_to_record_splitting_zone
return $end_lsn;
}
-
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
my $node = PostgreSQL::Test::Cluster->new("node");
$node->init;
-$node->append_conf('postgresql.conf',
- q[wal_level = minimal
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
autovacuum = off
checkpoint_timeout = '30min'
]);
$node->start;
-$node->safe_psql('postgres', "create table t as select 42");
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
-
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
my $end_lsn;
my $prev_lsn;
-my $log_size;
###########################################################################
note "Single-page end-of-WAL detection";
###########################################################################
-# xl_tot_len is 0 (a common case, we hit trailing zeroes)
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-$log_size = get_log_size($node);
+my $log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid record length at .*: expected at least 24, got 0",
- $log_size),
- "xl_tot_len zero");
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
-# xl_tot_len is < 24 (presumably recycled garbage)
+# xl_tot_len is < 24 (presumably recycled garbage).
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(23));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid record length at .*: expected at least 24, got 23",
- $log_size),
- "xl_tot_len short");
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
-# need more pages, but xl_prev check fails first
+# Need more pages, but xl_prev check fails first.
$end_lsn = advance_out_of_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("record with incorrect prev-link 0/DEADBEEF at .*",
- $log_size),
- "xl_prev bad");
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
-# xl_crc check fails
+# xl_crc check fails.
advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 10);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn - 8, '!'); # corrupt a byte in that record
-$log_size = get_log_size($node);
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("incorrect resource manager data checksum in record at .*",
- $log_size),
- "xl_crc bad");
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
###########################################################################
note "Multi-page end-of-WAL detection, header is not split";
###########################################################################
-# good xl_prev, we hit zero page next (zero magic)
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
$prev_lsn = advance_out_of_record_splitting_zone($node);
-#my $prev_lsn = emit_message($node, 0);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero");
+ "xlp_magic zero");
-# good xl_prev, we hit garbage page next (bad magic)
+# Good xl_prev, we hit garbage page next (bad magic).
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header(0xcafe,
- 0,
- 1,
- 0));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
- "xlp_magic bad");
+ "xlp_magic bad");
-# good xl_prev, we hit typical recycled page (good xlp_magic, bad xlp_pageaddr)
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0,
- 1,
- 0xbaaaaaad));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad");
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
-# good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0x1234,
- 1,
- start_of_next_page($end_lsn)));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
- "xlp_info bad");
+ "xlp_info bad");
-# good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention cont record
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- 0,
- 1,
- start_of_next_page($end_lsn)));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("there is no contrecord flag at .*", $log_size),
- "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
-# good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add up
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
$prev_lsn = advance_out_of_record_splitting_zone($node);
$end_lsn = emit_message($node, 0);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 42,
- $prev_lsn));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- start_of_next_page($end_lsn),
- 123456));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad");
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
###########################################################################
note "Multi-page, but header is split, so page checks are done first";
###########################################################################
-# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
$node->start;
ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero (split record header)");
+ "xlp_magic zero (split record header)");
-# and we'll also check xlp_pageaddr before any header checks
+# And we'll also check xlp_pageaddr before any header checks.
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- 0xbaaaaaad));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad (split record header)");
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
-# we'll also discover that xlp_rem_len doesn't add up before any
-# header checks
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
$end_lsn = advance_to_record_splitting_zone($node);
$node->stop('immediate');
-write_wal($node, 1, $end_lsn, record_header(2 * 1024 * 1024 * 1024,
- 0,
- 0xdeadbeef));
-write_wal($node, 1, start_of_next_page($end_lsn),
- page_header($XLP_PAGE_MAGIC,
- $XLP_FIRST_IS_CONTRECORD,
- 1,
- start_of_next_page($end_lsn),
- 123456));
-$log_size = get_log_size($node);
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
$node->start;
-ok($node->log_contains("invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad (split record header)");
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
done_testing();
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZPQdHY4WDR+2ejwq@paquier.xyz/5-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-03 11:21 Sergei Kornilov <sk@zsrv.org>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Sergei Kornilov @ 2023-09-03 11:21 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
Hello
Thank you, this version passes check-world for me.
Looks good (but I'm certainly not a big expert on xlogreader), a solid set of tests for different cases.
regards, Sergei
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-04 03:20 Thomas Munro <thomas.munro@gmail.com>
parent: Sergei Kornilov <sk@zsrv.org>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-04 03:20 UTC (permalink / raw)
To: Sergei Kornilov <sk@zsrv.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
Thanks for looking/testing, Sergei. Thanks for the changes, Michael,
these all look good. I've squashed them and added you as co-author.
A couple more small comment/text changes:
1. In the place where we fail to allocate memory for an oversized
record, I copied the comment about treating that as a "bogus data"
condition. I suspect that we will soon be converting that to a FATAL
error[1], and that'll need to be done in both places.
2. In this version of the commit message I said we'd only back-patch
to 15 for now. After sleeping on this for a week, I realised that the
reason I keep vacillating on that point is that I am not sure what
your plan is for the malloc-failure-means-end-of-wal policy ([1],
ancient code from 0ffe11abd3a). If we're going to fix that in master
only but let sleeping dogs lie in the back-branches, then it becomes
less important to go back further than 15 with THIS patch. But if you
want to be able to distinguish garbage from out-of-memory, and thereby
end-of-wal from a FATAL please-insert-more-RAM condition, I think
you'd really need this industrial strength validation in all affected
branches, and I'd have more work to do, right? The weak validation we
are fixing here is the *real* underlying problem going back many
years, right?
I also wondered about strengthening the validation of various things
like redo begin/end LSNs etc in these tests. But we can always
continue to improve all this later...
Here also is a version for 15 (and a CI run[2]), since we tweaked many
of the error messages between 15 and 16.
[1] https://www.postgresql.org/message-id/flat/ZMh/WV%2BCuknqePQQ%40paquier.xyz
[2] https://cirrus-ci.com/task/4533280897236992 (failed on some
unrelated pgbench test, reported in another thread)
Attachments:
[application/octet-stream] v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch15 (24.0K, ../../CA+hUKG+jFN64gKVGcmfNTekqWn3cemRx99-B8DDBaDyzWnfpkw@mail.gmail.com/2-v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch15)
download
[text/x-patch] v5-0001-Don-t-trust-unvalidated-xl_tot_len.patch (24.2K, ../../CA+hUKG+jFN64gKVGcmfNTekqWn3cemRx99-B8DDBaDyzWnfpkw@mail.gmail.com/3-v5-0001-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From d8410edb28b06eaf907211cd7ba3055bbf2ac284 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..9de34a2260 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..aa5eec7c03
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.39.2
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-04 03:54 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-04 03:54 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Sep 04, 2023 at 03:20:31PM +1200, Thomas Munro wrote:
> 1. In the place where we fail to allocate memory for an oversized
> record, I copied the comment about treating that as a "bogus data"
> condition. I suspect that we will soon be converting that to a FATAL
> error[1], and that'll need to be done in both places.
You mean for the two callers of XLogReadRecordAlloc(), even for the
case where !allow_oversized? Using a FATAL on non-FRONTEND would be
the quickest fix, indeed, but there are argument for standbys where we
could let these continue, as well. That would be an improvement over
the always-FATAL on OOM, of course.
> 2. In this version of the commit message I said we'd only back-patch
> to 15 for now. After sleeping on this for a week, I realised that the
> reason I keep vacillating on that point is that I am not sure what
> your plan is for the malloc-failure-means-end-of-wal policy ([1],
> ancient code from 0ffe11abd3a). If we're going to fix that in master
> only but let sleeping dogs lie in the back-branches, then it becomes
> less important to go back further than 15 with THIS patch.
For HEAD, my plan was to give more control to the callers of the WAL
reader, which is not something that could happen in back-branches
because it requires a few structural changes in the reporting the
reader is able to do. That's the standby vs crash-recovery behavior,
where we can make the startup process not FATAL all the time because
it should just loop forever if it cannot process a record. It does
not fix the root issue discussed here, of course, but it make the OOM
handling more friendly in some replay cases.
> But if you
> want to be able to distinguish garbage from out-of-memory, and thereby
> end-of-wal from a FATAL please-insert-more-RAM condition, I think
> you'd really need this industrial strength validation in all affected
> branches, and I'd have more work to do, right? The weak validation we
> are fixing here is the *real* underlying problem going back many
> years, right?
Getting the same validation checks for all the branches would be nice.
FATAL-ing on OOM to force recovery to happen again is a better option
than assuming that it is the end of recovery. I am OK to provide
patches for all the branches for the sake of this thread, if that
helps. Switching to a hard FATAL on OOM for the WAL reader in the
backend is backpatchable, but I'd rather consider that on a different
thread once the better checks for the record header are in place.
> I also wondered about strengthening the validation of various things
> like redo begin/end LSNs etc in these tests. But we can always
> continue to improve all this later...
Agreed.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZPVU9FwENbnO6Q++@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-04 07:17 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-04 07:17 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Sep 4, 2023 at 3:54 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Sep 04, 2023 at 03:20:31PM +1200, Thomas Munro wrote:
> > 1. In the place where we fail to allocate memory for an oversized
> > record, I copied the comment about treating that as a "bogus data"
> > condition. I suspect that we will soon be converting that to a FATAL
> > error[1], and that'll need to be done in both places.
>
> You mean for the two callers of XLogReadRecordAlloc(), even for the
> case where !allow_oversized? Using a FATAL on non-FRONTEND would be
> the quickest fix, indeed, but there are argument for standbys where we
> could let these continue, as well. That would be an improvement over
> the always-FATAL on OOM, of course.
I just mean the two places where "bogus data" is mentioned in that v5 patch.
> > But if you
> > want to be able to distinguish garbage from out-of-memory, and thereby
> > end-of-wal from a FATAL please-insert-more-RAM condition, I think
> > you'd really need this industrial strength validation in all affected
> > branches, and I'd have more work to do, right? The weak validation we
> > are fixing here is the *real* underlying problem going back many
> > years, right?
>
> Getting the same validation checks for all the branches would be nice.
> FATAL-ing on OOM to force recovery to happen again is a better option
> than assuming that it is the end of recovery. I am OK to provide
> patches for all the branches for the sake of this thread, if that
> helps. Switching to a hard FATAL on OOM for the WAL reader in the
> backend is backpatchable, but I'd rather consider that on a different
> thread once the better checks for the record header are in place.
OK, so it sounds like you want to go back to 12. Let me see if I can
get this TAP test to work in 12... more tomorrow.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-04 07:28 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-04 07:28 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Sep 04, 2023 at 07:17:21PM +1200, Thomas Munro wrote:
> I just mean the two places where "bogus data" is mentioned in that v5 patch.
Got it, thanks.
> OK, so it sounds like you want to go back to 12. Let me see if I can
> get this TAP test to work in 12... more tomorrow.
If you're restricted on time, I am OK to deep-dive into these branches
as well ;)
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZPWHHj0cbcKl1x7L@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-04 07:43 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-04 07:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Sep 4, 2023 at 7:28 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Sep 04, 2023 at 07:17:21PM +1200, Thomas Munro wrote:
> > OK, so it sounds like you want to go back to 12. Let me see if I can
> > get this TAP test to work in 12... more tomorrow.
>
> If you're restricted on time, I am OK to deep-dive into these branches
> as well ;)
I am behind on several things, so if you want to, please go ahead! I
already posted a 15 version and a master version (which should also
apply and work on 16), so we'd need 12, 13, 14.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-05 03:19 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-05 03:19 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
On Mon, Sep 04, 2023 at 07:43:59PM +1200, Thomas Munro wrote:
> I am behind on several things, so if you want to, please go ahead! I
> already posted a 15 version and a master version (which should also
> apply and work on 16), so we'd need 12, 13, 14.
And here you go for all the three branches, with all the tests
passing. This change is complicated enough that it would be better to
get more eyes on the three patches for these specific branches. Now,
this is much simpler since we don't have to worry about the
prefetching.
For these versions, it struck me that we *require* 6860198 so as the
header validation is able to correctly happen across page reads when
doing crash recovery. I have added this change in the patches
attached for simplicity, but 6860198 should be applied first on 12~14
as an independent change. I can do that as a first step, if you are
OK with that.
I haven't taken the time to rewrite the commit message yet for the
main patch ;p
--
Michael
From c2bb94cf08b2bc472fd5d73270a4c09704f78d57 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 36 ++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 539 insertions(+), 41 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dbd72c0a16..98e4cbc27d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12041,7 +12041,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12064,9 +12064,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 26f8e0ae2c..0016a3b3ef 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -155,6 +155,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -164,25 +167,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -346,15 +330,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -370,17 +346,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -475,8 +445,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index de851e1615..5f80ff9cbc 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -33,6 +33,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -459,6 +460,41 @@ sub chmod_recursive
return;
}
+# Returns an array that stores all the matches of the given regular expression
+# within the PostgreSQL installation's C<header_path>. This can be used to
+# retrieve specific value patterns from the installation's header files.
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
# Check presence of a given regexp within pg_config.h for the installation
# where tests are running, returning a match status result depending on
# that.
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From e4c6758ce6a4643db87949e9661ef6adf79a80e7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 546 insertions(+), 41 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 885a12a114..04d23bfe4a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12299,7 +12299,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12322,9 +12322,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 911bc91bc9..f2b268d236 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,17 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +495,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7715d551b3..fee244c418 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -66,6 +66,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -632,6 +633,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From 5810e7788999d96bb7a1cb1adc0cd230f057436e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 75 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 545 insertions(+), 40 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index a5d56690ae..1f139fa743 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12525,7 +12525,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12549,9 +12549,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d797d9d508..9d74507dbe 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,16 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +496,36 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7860425b54..307090321f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -70,6 +70,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -648,6 +649,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..c918cc31cb
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
Attachments:
[text/plain] v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch12 (23.5K, ../../ZPaeUimGAzhYVEn9@paquier.xyz/2-v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch12)
download | inline diff:
From c2bb94cf08b2bc472fd5d73270a4c09704f78d57 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 36 ++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 539 insertions(+), 41 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dbd72c0a16..98e4cbc27d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12041,7 +12041,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12064,9 +12064,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 26f8e0ae2c..0016a3b3ef 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -155,6 +155,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -164,25 +167,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -346,15 +330,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -370,17 +346,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -475,8 +445,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index de851e1615..5f80ff9cbc 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -33,6 +33,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -459,6 +460,41 @@ sub chmod_recursive
return;
}
+# Returns an array that stores all the matches of the given regular expression
+# within the PostgreSQL installation's C<header_path>. This can be used to
+# retrieve specific value patterns from the installation's header files.
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
# Check presence of a given regexp within pg_config.h for the installation
# where tests are running, returning a match status result depending on
# that.
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch13 (23.5K, ../../ZPaeUimGAzhYVEn9@paquier.xyz/3-v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch13)
download | inline diff:
From e4c6758ce6a4643db87949e9661ef6adf79a80e7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 546 insertions(+), 41 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 885a12a114..04d23bfe4a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12299,7 +12299,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12322,9 +12322,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 911bc91bc9..f2b268d236 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,17 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +495,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7715d551b3..fee244c418 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -66,6 +66,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -632,6 +633,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch14 (23.4K, ../../ZPaeUimGAzhYVEn9@paquier.xyz/4-v5-0001-Don-t-trust-unvalidated-xl_tot_len.backpatch14)
download | inline diff:
From 5810e7788999d96bb7a1cb1adc0cd230f057436e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v5] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlog.c | 18 +-
src/backend/access/transam/xlogreader.c | 75 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
4 files changed, 545 insertions(+), 40 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index a5d56690ae..1f139fa743 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -12525,7 +12525,7 @@ retry:
/*
* Check the page header immediately, so that we can retry immediately if
- * it's not valid. This may seem unnecessary, because XLogReadRecord()
+ * it's not valid. This may seem unnecessary, because ReadPageInternal()
* validates the page header anyway, and would propagate the failure up to
* ReadRecord(), which would retry. However, there's a corner case with
* continuation records, if a record is split across two pages such that
@@ -12549,9 +12549,23 @@ retry:
*
* Validating the page header is cheap enough that doing it twice
* shouldn't be a big deal from a performance point of view.
+ *
+ * When not in standby mode, an invalid page header should cause recovery
+ * to end, not retry reading the page, so we don't need to validate the
+ * page header here for the retry. Instead, ReadPageInternal() is
+ * responsible for the validation.
*/
- if (!XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
+ if (StandbyMode &&
+ !XLogReaderValidatePageHeader(xlogreader, targetPagePtr, readBuf))
{
+ /*
+ * Emit this error right now then retry this page immediately. Use
+ * errmsg_internal() because the message was already translated.
+ */
+ if (xlogreader->errormsg_buf[0])
+ ereport(emode_for_corrupt_record(emode, EndRecPtr),
+ (errmsg_internal("%s", xlogreader->errormsg_buf)));
+
/* reset any error XLogReaderValidatePageHeader() might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d797d9d508..9d74507dbe 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,16 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +496,36 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7860425b54..307090321f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -70,6 +70,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -648,6 +649,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..c918cc31cb
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZPaeUimGAzhYVEn9@paquier.xyz/5-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-05 03:58 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-05 03:58 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 5, 2023 at 3:19 PM Michael Paquier <michael@paquier.xyz> wrote:
> And here you go for all the three branches, with all the tests
> passing. This change is complicated enough that it would be better to
> get more eyes on the three patches for these specific branches. Now,
> this is much simpler since we don't have to worry about the
> prefetching.
Thanks! Yeah.
> For these versions, it struck me that we *require* 6860198 so as the
> header validation is able to correctly happen across page reads when
> doing crash recovery. I have added this change in the patches
> attached for simplicity, but 6860198 should be applied first on 12~14
> as an independent change. I can do that as a first step, if you are
> OK with that.
Ahhhh, that explains why back-patching was unexpectedly unsuccessful
when I tried it. I have added Fujii-san to the CC list. I don't see
any problem with that plan but it would be good to hear from him.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-05 04:13 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-05 04:13 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 05, 2023 at 03:58:42PM +1200, Thomas Munro wrote:
> Ahhhh, that explains why back-patching was unexpectedly unsuccessful
> when I tried it. I have added Fujii-san to the CC list. I don't see
> any problem with that plan but it would be good to hear from him.
Right. Let's wait for a bit and see if there are any comments on this
point. With 16.0 in sight next week, I'd like to propose that we move
on with this issue once next week's tag is pushed.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZPaq3KwARVHWzune@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-12 01:01 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-12 01:01 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 05, 2023 at 01:13:16PM +0900, Michael Paquier wrote:
> Right. Let's wait for a bit and see if there are any comments on this
> point. With 16.0 in sight next week, I'd like to propose that we move
> on with this issue once next week's tag is pushed.
Hearing nothing, I have applied the backpatch of 0668719801 to 12~14.
Please find attached the remaining patches for all the stable
branches.
Thomas, would you prefer handling that yourself once 16.0 is tagged
with a last round of checks? Or should I? (Still need to edit the
commit message a bit..)
--
Michael
From ea7a54c827f18b56796189c0e9d36224e6a32a84 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 443 ++++++++++++++++++++++++
4 files changed, 554 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..9de34a2260 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..aa5eec7c03
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From 8a75b763a8c477d1cb62a7e52bb825894a3bcdce Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 443 ++++++++++++++++++++++++
3 files changed, 553 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 03fad82bc1..4b48abfa05 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -667,41 +651,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -716,16 +685,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -822,8 +786,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -865,6 +856,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -893,8 +906,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 4e18730dd0..4fd2e48ca9 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -69,6 +69,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -672,6 +673,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..779a195a7d
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From a71a7beb88405e24ee9dd3bc616dc26808845bbe Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 75 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 529 insertions(+), 38 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d797d9d508..9d74507dbe 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,16 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +496,36 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7860425b54..307090321f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -70,6 +70,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -648,6 +649,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..c918cc31cb
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From 9156107d5e3ae860d332cdb3af581ba4c314844d Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 530 insertions(+), 39 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 911bc91bc9..f2b268d236 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,17 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +495,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7715d551b3..fee244c418 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -66,6 +66,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -632,6 +633,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
From ae0864bebe0e475f1a1ee04c2ada573c5e861e38 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 36 ++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 523 insertions(+), 39 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 26f8e0ae2c..0016a3b3ef 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -155,6 +155,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -164,25 +167,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -346,15 +330,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -370,17 +346,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -475,8 +445,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index de851e1615..5f80ff9cbc 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -33,6 +33,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -459,6 +460,41 @@ sub chmod_recursive
return;
}
+# Returns an array that stores all the matches of the given regular expression
+# within the PostgreSQL installation's C<header_path>. This can be used to
+# retrieve specific value patterns from the installation's header files.
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
# Check presence of a given regexp within pg_config.h for the installation
# where tests are running, returning a match status result depending on
# that.
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
Attachments:
[text/x-diff] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch (24.5K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/2-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From 5a294edd527c87248e5c0e47dfdb8a5d43944afe Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 443 ++++++++++++++++++++++++
4 files changed, 554 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..9de34a2260 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..aa5eec7c03
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16 (24.5K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/3-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16)
download | inline diff:
From ea7a54c827f18b56796189c0e9d36224e6a32a84 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_end_of_wal.pl | 443 ++++++++++++++++++++++++
4 files changed, 554 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..9de34a2260 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..10407668d4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..aa5eec7c03
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15 (24.0K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/4-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15)
download | inline diff:
From 8a75b763a8c477d1cb62a7e52bb825894a3bcdce Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 443 ++++++++++++++++++++++++
3 files changed, 553 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 03fad82bc1..4b48abfa05 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -667,41 +651,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -716,16 +685,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -822,8 +786,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -865,6 +856,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -893,8 +906,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 4e18730dd0..4fd2e48ca9 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -69,6 +69,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -672,6 +673,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..779a195a7d
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,443 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14 (21.7K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/5-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14)
download | inline diff:
From a71a7beb88405e24ee9dd3bc616dc26808845bbe Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 75 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 529 insertions(+), 38 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index d797d9d508..9d74507dbe 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,16 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +496,36 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7860425b54..307090321f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -70,6 +70,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -648,6 +649,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..c918cc31cb
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13 (21.8K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/6-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13)
download | inline diff:
From 9156107d5e3ae860d332cdb3af581ba4c314844d Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 43 +++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 530 insertions(+), 39 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 911bc91bc9..f2b268d236 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -168,6 +168,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -177,25 +180,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -396,15 +380,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -420,17 +396,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -525,8 +495,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 7715d551b3..fee244c418 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -66,6 +66,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -632,6 +633,48 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[text/plain] v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12 (21.8K, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/7-v6-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12)
download | inline diff:
From ae0864bebe0e475f1a1ee04c2ada573c5e861e38 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 14 Aug 2023 17:06:42 +1200
Subject: [PATCH v6] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We should fix that too, but that wouldn't protect us from
rare crash scenarios where the trailing zeroes don't make it to disk.
Back-patch to 15. We might decide to back-patch a variant of this
to older branches with more study, but the symptoms there (bogus large
allocation) are less severe.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 77 ++--
src/test/perl/TestLib.pm | 36 ++
src/test/recovery/t/038_end_of_wal.pl | 449 ++++++++++++++++++++++++
3 files changed, 523 insertions(+), 39 deletions(-)
create mode 100644 src/test/recovery/t/038_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 26f8e0ae2c..0016a3b3ef 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -155,6 +155,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -164,25 +167,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -346,15 +330,7 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: wanted %u, got %u",
- (uint32) (RecPtr >> 32), (uint32) RecPtr,
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
@@ -370,17 +346,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len,
- (uint32) (RecPtr >> 32), (uint32) RecPtr);
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -475,8 +445,37 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ Assert(gotheader);
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len,
+ (uint32) (RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index de851e1615..5f80ff9cbc 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -33,6 +33,7 @@ our @EXPORT = qw(
check_mode_recursive
chmod_recursive
check_pg_config
+ scan_server_header
system_or_bail
system_log
run_log
@@ -459,6 +460,41 @@ sub chmod_recursive
return;
}
+# Returns an array that stores all the matches of the given regular expression
+# within the PostgreSQL installation's C<header_path>. This can be used to
+# retrieve specific value patterns from the installation's header files.
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if ($line =~ /^$regexp/)
+ {
+ # Found match, so store all the results.
+ @match = @{^CAPTURE};
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
# Check presence of a given regexp within pg_config.h for the installation
# where tests are running, returning a match status result depending on
# that.
diff --git a/src/test/recovery/t/038_end_of_wal.pl b/src/test/recovery/t/038_end_of_wal.pl
new file mode 100644
index 0000000000..6927d1298a
--- /dev/null
+++ b/src/test/recovery/t/038_end_of_wal.pl
@@ -0,0 +1,449 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end of WAL replay. This test suite generates fake
+# WAL records able to trigger various failure scenarios at replay.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at offset.
+# This should be called while the cluster is offline.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+sub format_lsn
+{
+ my $lsn = shift;
+ return sprintf("%X/%X", $lsn >> 32, $lsn & 0xffffffff);
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = 2000;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: wanted 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE ", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 ", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 ", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD ", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZP+4YYrPZk0GFlc3@paquier.xyz/8-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-12 02:08 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-12 02:08 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 12, 2023 at 1:01 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Sep 05, 2023 at 01:13:16PM +0900, Michael Paquier wrote:
> > Right. Let's wait for a bit and see if there are any comments on this
> > point. With 16.0 in sight next week, I'd like to propose that we move
> > on with this issue once next week's tag is pushed.
>
> Hearing nothing, I have applied the backpatch of 0668719801 to 12~14.
> Please find attached the remaining patches for all the stable
> branches.
Thanks!
> Thomas, would you prefer handling that yourself once 16.0 is tagged
> with a last round of checks? Or should I? (Still need to edit the
> commit message a bit..)
I'll do that (and update the message). Thanks for all the help!
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-12 02:12 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-12 02:12 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 12, 2023 at 02:08:09PM +1200, Thomas Munro wrote:
> I'll do that (and update the message). Thanks for all the help!
Cool, thanks!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZP%2FI93ZSL0Mvt5rS@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-15 22:07 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-15 22:07 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 12, 2023 at 2:12 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Sep 12, 2023 at 02:08:09PM +1200, Thomas Munro wrote:
> > I'll do that (and update the message). Thanks for all the help!
>
> Cool, thanks!
I rebased (bumped the test number 038 -> 039), updated the commit
message[1], tested on a big endian machine (AIX/POWER), but then I
found there is something not quite right or stable with the WAL
positioning logic. On one of my machines, but not another, the
back-patch to 12 fails in one of the tests unless I hack it to insert
a bit more WAL, and I think that's telling me that Perl code to
control the insertion point is not quite bullet proof enough yet.
I'll need to dig into this some more on Monday.
[1] https://github.com/macdice/postgres/commits/fix-12
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-16 00:02 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-16 00:02 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 16, 2023 at 10:07:30AM +1200, Thomas Munro wrote:
> I rebased (bumped the test number 038 -> 039), updated the commit
> message[1], tested on a big endian machine (AIX/POWER), but then I
> found there is something not quite right or stable with the WAL
> positioning logic. On one of my machines, but not another, the
> back-patch to 12 fails in one of the tests unless I hack it to insert
> a bit more WAL, and I think that's telling me that Perl code to
> control the insertion point is not quite bullet proof enough yet.
> I'll need to dig into this some more on Monday.
>
> [1] https://github.com/macdice/postgres/commits/fix-12
Hmm. What was the test that failed?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQTwsT6tsbx6BYn3@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-16 00:20 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 86+ messages in thread
From: Thomas Munro @ 2023-09-16 00:20 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 16, 2023 at 12:03 PM Michael Paquier <michael@paquier.xyz> wrote:
> > [1] https://github.com/macdice/postgres/commits/fix-12
>
> Hmm. What was the test that failed?
$ make -s -C src/test/recovery/ check PROVE_TESTS=t/039*
t/039_end_of_wal.pl .. 4/?
# Failed test 'xlp_magic zero'
# at t/039_end_of_wal.pl line 312.
not ok 5 - xlp_magic zero
Where the log should say "invalid magic number 0000" I see:
2023-09-16 12:13:07.331 NZST [156812] LOG: record with incorrect
prev-link 0/16B60C0 at 0/16B6120
It has to do with initial WAL position after initdb, because I get
this only on Debian, on REL_12_STABLE (with the commit listed above on
my public fix-12 branch) and only with --with-icu, but not without it,
and I can't repro it on my other local OSes.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-16 04:13 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2023-09-16 04:13 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; exclusion@gmail.com, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 16, 2023 at 12:20:36PM +1200, Thomas Munro wrote:
> It has to do with initial WAL position after initdb, because I get
> this only on Debian, on REL_12_STABLE (with the commit listed above on
> my public fix-12 branch) and only with --with-icu, but not without it,
> and I can't repro it on my other local OSes.
That's close to my dev environment, with SID and x64 and the only
difference your branch and my local branch is a rename of the TAP
file, and I cannot see the problem.
Some random thoughts: perhaps reducing $page_threshold would help with
more records inserted to get to the limit of a page, reducing noise?
Or perhaps skipping a few pages would?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQUrbfwQVr417W0v@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-16 06:00 Alexander Lakhin <exclusion@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 2 replies; 86+ messages in thread
From: Alexander Lakhin @ 2023-09-16 06:00 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
Hello,
16.09.2023 03:20, Thomas Munro wrote:
> On Sat, Sep 16, 2023 at 12:03 PM Michael Paquier <michael@paquier.xyz> wrote:
>>> [1] https://github.com/macdice/postgres/commits/fix-12
>> Hmm. What was the test that failed?
> $ make -s -C src/test/recovery/ check PROVE_TESTS=t/039*
> t/039_end_of_wal.pl .. 4/?
> # Failed test 'xlp_magic zero'
> # at t/039_end_of_wal.pl line 312.
>
> not ok 5 - xlp_magic zero
>
> Where the log should say "invalid magic number 0000" I see:
>
> 2023-09-16 12:13:07.331 NZST [156812] LOG: record with incorrect
> prev-link 0/16B60C0 at 0/16B6120
>
> It has to do with initial WAL position after initdb, because I get
> this only on Debian, on REL_12_STABLE (with the commit listed above on
> my public fix-12 branch) and only with --with-icu, but not without it,
> and I can't repro it on my other local OSes.
I tried to reproduce the failure on Debian 9, 10, 11, but not succeeded yet.
Though I got another error on Debian 9:
t/039_end_of_wal.pl .. Dubious, test returned 25 (wstat 6400, 0x1900)
No subtests run
...
cat src/test/recovery/tmp_check/log/regress_log_039_end_of_wal
could not find match in header access/xlog_internal.h
It looks like the construction "@{^CAPTURE}" used in scan_server_header()
is not supported by Perl 5.24, which is included in Debian stretch:
https://perldoc.perl.org/variables/@%7B%5ECAPTURE%7D
I replaced it with
@match = ($1);
and that worked for me.
Also, I observed that "wal_log_hints = on" in extra.config, which I use via
"TEMP_CONFIG=extra.config make check-world" makes the test fail too, though
check-world passes fine without the new test.
Maybe that's not an issue, and probably there are other parameters, which
might affect this test, but I'm somewhat confused by the fact that only this
test breaks with it.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-17 20:46 Thomas Munro <thomas.munro@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
1 sibling, 0 replies; 86+ messages in thread
From: Thomas Munro @ 2023-09-17 20:46 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 16, 2023 at 6:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> It looks like the construction "@{^CAPTURE}" used in scan_server_header()
> is not supported by Perl 5.24, which is included in Debian stretch:
> https://perldoc.perl.org/variables/@%7B%5ECAPTURE%7D
> I replaced it with
> @match = ($1);
> and that worked for me.
Thanks for catching that! I see that %{^CAPTURE} was added to Perl
5.25.7, but we document Perl 5.14 or later as the requirement
currently, and for REL_12_STABLE it's Perl 5.8.3. That would
certainly have turned a few build farm fossils red.
Your replacement only returns the first captured group. I don't know
Perl, but I think we can just assign the whole array to @match
directly, to maintain the semantics that Michael wanted, like so:
- if ($line =~ /^$regexp/)
+ if (@match = $line =~ /^$regexp/)
{
- # Found match, so store all the results.
- @match = @{^CAPTURE};
last;
}
Better Perl welcome.
Looking into your observation about wal_log_hints...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-19 09:25 Thomas Munro <thomas.munro@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
1 sibling, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-19 09:25 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 16, 2023 at 6:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> Also, I observed that "wal_log_hints = on" in extra.config, which I use via
> "TEMP_CONFIG=extra.config make check-world" makes the test fail too, though
> check-world passes fine without the new test.
> Maybe that's not an issue, and probably there are other parameters, which
> might affect this test, but I'm somewhat confused by the fact that only this
> test breaks with it.
The problem was that the Perl functions advance_XXX() would sometimes
see an "unflushed" insert position caused by hint records and other
incidental stuff, and then we'd write our contrived header at a
disconnected location after a bunch of (unwritten) zeroes. The
solution in this version is to start each test with
emit_message($node, 0), which flushes a small record.
Attachments:
[text/x-patch] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch (25.1K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/2-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From 9d3cdf3bc1756e81f74136b25eb64cd303251a9b Mon Sep 17 00:00:00 2001
From: Thomas Munro <tmunro@postgresql.org>
Date: Fri, 15 Sep 2023 11:32:36 +1200
Subject: [PATCH v7] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We could fix that too, but it wouldn't protect us from rare
crash scenarios where the trailing zeroes don't make it to disk.
With reliable xl_tot_len validation in place, the ancient policy of
considering malloc failure to indicate corruption at end-of-WAL seems
quite surprising, but changing that is left for later work.
Also included is a new TAP test to exercise various cases of end-of-WAL
detection by writing contrived data into the WAL from Perl.
Back-patch to 12. We decided not to put this change into the final
release of 11.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 41 +++
src/test/recovery/meson.build | 1 +
src/test/recovery/t/039_end_of_wal.pl | 457 ++++++++++++++++++++++++
4 files changed, 566 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/039_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..d66fe1cf45 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,46 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if (@match = $line =~ /^$regexp/)
+ {
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 646d6ffde4..9d8039684a 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -44,6 +44,7 @@ tests += {
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
't/038_save_logical_slots_shutdown.pl',
+ 't/039_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl
new file mode 100644
index 0000000000..bd666bc078
--- /dev/null
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -0,0 +1,457 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end-of-WAL conditions. This test suite generates
+# fake defective page and record headers to trigger various failure
+# scenarios.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at LSN.
+# This should be called while the cluster is not running.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ $end_lsn = emit_message($node, $page_threshold);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ $end_lsn = emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ $end_lsn = emit_message($node, $message_size);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+emit_message($node, 0);
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xlp_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.41.0
[application/octet-stream] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16 (25.1K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/3-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16)
download
[application/octet-stream] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15 (24.5K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/4-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15)
download
[application/octet-stream] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14 (22.2K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/5-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14)
download
[application/octet-stream] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13 (22.3K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/6-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13)
download
[application/octet-stream] v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12 (22.4K, ../../CA+hUKGK53D8OYxTQiQ9qzJsA7ymcHTqqh7V_PUM06TCT=HOJqg@mail.gmail.com/7-v7-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-19 09:53 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-19 09:53 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 19, 2023 at 09:25:34PM +1200, Thomas Munro wrote:
> The problem was that the Perl functions advance_XXX() would sometimes
> see an "unflushed" insert position caused by hint records and other
> incidental stuff, and then we'd write our contrived header at a
> disconnected location after a bunch of (unwritten) zeroes. The
> solution in this version is to start each test with
> emit_message($node, 0), which flushes a small record.
Makes sense. Does this also address the other issue you've noticed on
v12 when ICU is enabled (the one with the initial LSN changed after
initdb)?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQlvihvHo5JspVpV@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-19 10:09 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-19 10:09 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Tue, Sep 19, 2023 at 9:53 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Sep 19, 2023 at 09:25:34PM +1200, Thomas Munro wrote:
> > The problem was that the Perl functions advance_XXX() would sometimes
> > see an "unflushed" insert position caused by hint records and other
> > incidental stuff, and then we'd write our contrived header at a
> > disconnected location after a bunch of (unwritten) zeroes. The
> > solution in this version is to start each test with
> > emit_message($node, 0), which flushes a small record.
>
> Makes sense. Does this also address the other issue you've noticed on
> v12 when ICU is enabled (the one with the initial LSN changed after
> initdb)?
Yep.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-19 13:00 Alexander Lakhin <exclusion@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Alexander Lakhin @ 2023-09-19 13:00 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Michael Paquier <michael@paquier.xyz>; +Cc: Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
19.09.2023 13:09, Thomas Munro wrote:
> On Tue, Sep 19, 2023 at 9:53 PM Michael Paquier<michael@paquier.xyz> wrote:
>> On Tue, Sep 19, 2023 at 09:25:34PM +1200, Thomas Munro wrote:
>>> The problem was that the Perl functions advance_XXX() would sometimes
>>> see an "unflushed" insert position caused by hint records and other
>>> incidental stuff, and then we'd write our contrived header at a
>>> disconnected location after a bunch of (unwritten) zeroes. The
>>> solution in this version is to start each test with
>>> emit_message($node, 0), which flushes a small record.
>>>
Thanks for the fixes!
I've tested this patch version on Debian 9 and it works fine now.
But as to wal_log_hints, I see that v7 patches work for REL_12_STABLE and
REL_13_STABLE, but not for other branches, though v6 patches definitely
failed on REL_12_STABLE and REL_13_STABLE too. So the behavior improved,
but not for all versions, for some reason (I haven't looked deeper yet).
Also I've noticed a few minor discrepancies in the patches.
A commit message in patch.15 contains the sentence "We should fix ...",
not "We could fix ...", as in commit messages for other versions.
Also, patch.12 contains a duplicated "Reviewed-by" string.
And may be "xlp_prev" is a typo?
Best regards,
Alexander
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-19 22:51 Thomas Munro <thomas.munro@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 2 replies; 86+ messages in thread
From: Thomas Munro @ 2023-09-19 22:51 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Wed, Sep 20, 2023 at 1:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> I've tested this patch version on Debian 9 and it works fine now.
Progress...
> But as to wal_log_hints, I see that v7 patches work for REL_12_STABLE and
> REL_13_STABLE, but not for other branches, though v6 patches definitely
> failed on REL_12_STABLE and REL_13_STABLE too. So the behavior improved,
> but not for all versions, for some reason (I haven't looked deeper yet).
Ahh, yeah, there was a second problem screwing up the LSN accounting.
I was trying to write correct xl_prev links, but also emitting
optional padding records to get into the right position (depending on
initial conditions varying between branches etc), and I forgot about
the extra COMMIT records that emit_message() generates. Which, I
guess, comes back to Michael's observation that this would all be a
bit easier if we had a way to emit and flush a single record...
The solution in this version is to call get_insert_len() instead of
using the result of emit_message() for the values returned by the
advance_XXX() functions. The result of emit_message() is actually the
LSN of the following COMMIT record so can't be used directly for
building xl_prev chains.
> Also I've noticed a few minor discrepancies in the patches.
> A commit message in patch.15 contains the sentence "We should fix ...",
> not "We could fix ...", as in commit messages for other versions.
> Also, patch.12 contains a duplicated "Reviewed-by" string.
> And may be "xlp_prev" is a typo?
Fixed.
Attachments:
[text/x-patch] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch (25.1K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/2-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch)
download | inline diff:
From ea03a850af0466d4701dd82ad87b17f6e0f907be Mon Sep 17 00:00:00 2001
From: Thomas Munro <tmunro@postgresql.org>
Date: Fri, 15 Sep 2023 11:32:36 +1200
Subject: [PATCH v8] Don't trust unvalidated xl_tot_len.
xl_tot_len comes first in a WAL record. Usually we don't trust it to be
the true length until we've validated the record header. If the record
header was split across two pages, previously we wouldn't do the
validation until after we'd already tried to allocate enough memory to
hold the record, which was bad because it might actually be garbage
bytes from a recycled WAL file, so we could try to allocate a lot of
memory. Release 15 made it worse.
Since 70b4f82a4b5, we'd at least generate an end-of-WAL condition if the
garbage 4 byte value happened to be > 1GB, but we'd still try to
allocate up to 1GB of memory bogusly otherwise. That was an
improvement, but unfortunately release 15 tries to allocate another
object before that, so you could get a FATAL error and recovery could
fail.
We can fix both variants of the problem more fundamentally using
pre-existing page-level validation, if we just re-order some logic.
The new order of operations in the split-header case defers all memory
allocation based on xl_tot_len until we've read the following page. At
that point we know that its first few bytes are not recycled data, by
checking its xlp_pageaddr, and that its xlp_rem_len agrees with
xl_tot_len on the preceding page. That is strong evidence that
xl_tot_len was truly the start of a record that was logged.
This problem was most likely to occur on a standby, because
walreceiver.c recycles WAL files without zeroing out trailing regions of
each page. We could fix that too, but it wouldn't protect us from rare
crash scenarios where the trailing zeroes don't make it to disk.
With reliable xl_tot_len validation in place, the ancient policy of
considering malloc failure to indicate corruption at end-of-WAL seems
quite surprising, but changing that is left for later work.
Also included is a new TAP test to exercise various cases of end-of-WAL
detection by writing contrived data into the WAL from Perl.
Back-patch to 12. We decided not to put this change into the final
release of 11.
Author: Thomas Munro <thomas.munro@gmail.com>
Author: Michael Paquier <michael@paquier.xyz>
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com> (the idea, not the code)
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Sergei Kornilov <sk@zsrv.org>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17928-aa92416a70ff44a2%40postgresql.org
---
src/backend/access/transam/xlogreader.c | 123 ++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 41 +++
src/test/recovery/meson.build | 1 +
src/test/recovery/t/039_end_of_wal.pl | 460 ++++++++++++++++++++++++
4 files changed, 569 insertions(+), 56 deletions(-)
create mode 100644 src/test/recovery/t/039_end_of_wal.pl
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c9f9f6e98f..58654b746c 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -192,6 +192,9 @@ XLogReaderFree(XLogReaderState *state)
* XLOG_BLCKSZ, and make sure it's at least 5*Max(BLCKSZ, XLOG_BLCKSZ) to start
* with. (That is enough for all "normal" records, but very large commit or
* abort records might need more space.)
+ *
+ * Note: This routine should *never* be called for xl_tot_len until the header
+ * of the record has been fully validated.
*/
static bool
allocate_recordbuf(XLogReaderState *state, uint32 reclength)
@@ -201,25 +204,6 @@ allocate_recordbuf(XLogReaderState *state, uint32 reclength)
newSize += XLOG_BLCKSZ - (newSize % XLOG_BLCKSZ);
newSize = Max(newSize, 5 * Max(BLCKSZ, XLOG_BLCKSZ));
-#ifndef FRONTEND
-
- /*
- * Note that in much unlucky circumstances, the random data read from a
- * recycled segment can cause this routine to be called with a size
- * causing a hard failure at allocation. For a standby, this would cause
- * the instance to stop suddenly with a hard failure, preventing it to
- * retry fetching WAL from one of its sources which could allow it to move
- * on with replay without a manual restart. If the data comes from a past
- * recycled segment and is still valid, then the allocation may succeed
- * but record checks are going to fail so this would be short-lived. If
- * the allocation fails because of a memory shortage, then this is not a
- * hard failure either per the guarantee given by MCXT_ALLOC_NO_OOM.
- */
- if (!AllocSizeIsValid(newSize))
- return false;
-
-#endif
-
if (state->readRecordBuf)
pfree(state->readRecordBuf);
state->readRecordBuf =
@@ -669,41 +653,26 @@ restart:
}
else
{
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
- "invalid record length at %X/%X: expected at least %u, got %u",
- LSN_FORMAT_ARGS(RecPtr),
- (uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
}
/*
- * Find space to decode this record. Don't allow oversized allocation if
- * the caller requested nonblocking. Otherwise, we *have* to try to
- * decode the record now because the caller has nothing else to do, so
- * allow an oversized record to be palloc'd if that turns out to be
- * necessary.
+ * Try to find space to decode this record, if we can do so without
+ * calling palloc. If we can't, we'll try again below after we've
+ * validated that total_len isn't garbage bytes from a recycled WAL page.
*/
decoded = XLogReadRecordAlloc(state,
total_len,
- !nonblocking /* allow_oversized */ );
- if (decoded == NULL)
+ false /* allow_oversized */ );
+ if (decoded == NULL && nonblocking)
{
/*
- * There is no space in the decode buffer. The caller should help
- * with that problem by consuming some records.
+ * There is no space in the circular decode buffer, and the caller is
+ * only reading ahead. The caller should consume existing records to
+ * make space.
*/
- if (nonblocking)
- return XLREAD_WOULDBLOCK;
-
- /* We failed to allocate memory for an oversized record. */
- report_invalid_record(state,
- "out of memory while trying to decode a record of length %u", total_len);
- goto err;
+ return XLREAD_WOULDBLOCK;
}
len = XLOG_BLCKSZ - RecPtr % XLOG_BLCKSZ;
@@ -718,16 +687,11 @@ restart:
assembled = true;
/*
- * Enlarge readRecordBuf as needed.
+ * We always have space for a couple of pages, enough to validate a
+ * boundary-spanning record header.
*/
- if (total_len > state->readRecordBufSize &&
- !allocate_recordbuf(state, total_len))
- {
- /* We treat this as a "bogus data" condition */
- report_invalid_record(state, "record length %u at %X/%X too long",
- total_len, LSN_FORMAT_ARGS(RecPtr));
- goto err;
- }
+ Assert(state->readRecordBufSize >= XLOG_BLCKSZ * 2);
+ Assert(state->readRecordBufSize >= len);
/* Copy the first fragment of the record from the first page. */
memcpy(state->readRecordBuf,
@@ -824,8 +788,35 @@ restart:
goto err;
gotheader = true;
}
- } while (gotlen < total_len);
+ /*
+ * We might need a bigger buffer. We have validated the record
+ * header, in the case that it split over a page boundary. We've
+ * also cross-checked total_len against xlp_rem_len on the second
+ * page, and verified xlp_pageaddr on both.
+ */
+ if (total_len > state->readRecordBufSize)
+ {
+ char save_copy[XLOG_BLCKSZ * 2];
+
+ /*
+ * Save and restore the data we already had. It can't be more
+ * than two pages.
+ */
+ Assert(gotlen <= lengthof(save_copy));
+ Assert(gotlen <= state->readRecordBufSize);
+ memcpy(save_copy, state->readRecordBuf, gotlen);
+ if (!allocate_recordbuf(state, total_len))
+ {
+ /* We treat this as a "bogus data" condition */
+ report_invalid_record(state, "record length %u at %X/%X too long",
+ total_len, LSN_FORMAT_ARGS(RecPtr));
+ goto err;
+ }
+ memcpy(state->readRecordBuf, save_copy, gotlen);
+ buffer = state->readRecordBuf + gotlen;
+ }
+ } while (gotlen < total_len);
Assert(gotheader);
record = (XLogRecord *) state->readRecordBuf;
@@ -867,6 +858,28 @@ restart:
state->NextRecPtr -= XLogSegmentOffset(state->NextRecPtr, state->segcxt.ws_segsize);
}
+ /*
+ * If we got here without a DecodedXLogRecord, it means we needed to
+ * validate total_len before trusting it, but by now now we've done that.
+ */
+ if (decoded == NULL)
+ {
+ Assert(!nonblocking);
+ decoded = XLogReadRecordAlloc(state,
+ total_len,
+ true /* allow_oversized */ );
+ if (decoded == NULL)
+ {
+ /*
+ * We failed to allocate memory for an oversized record. As
+ * above, we currently treat this as a "bogus data" condition.
+ */
+ report_invalid_record(state,
+ "out of memory while trying to decode a record of length %u", total_len);
+ goto err;
+ }
+ }
+
if (DecodeXLogRecord(state, decoded, record, RecPtr, &errormsg))
{
/* Record the location of the next record. */
@@ -895,8 +908,6 @@ restart:
state->decode_queue_head = decoded;
return XLREAD_SUCCESS;
}
- else
- return XLREAD_FAIL;
err:
if (assembled)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 617caa022f..d66fe1cf45 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -71,6 +71,7 @@ our @EXPORT = qw(
chmod_recursive
check_pg_config
dir_symlink
+ scan_server_header
system_or_bail
system_log
run_log
@@ -702,6 +703,46 @@ sub chmod_recursive
=pod
+=item scan_server_header(header_path, regexp)
+
+Returns an array that stores all the matches of the given regular expression
+within the PostgreSQL installation's C<header_path>. This can be used to
+retrieve specific value patterns from the installation's header files.
+
+=cut
+
+sub scan_server_header
+{
+ my ($header_path, $regexp) = @_;
+
+ my ($stdout, $stderr);
+ my $result = IPC::Run::run [ 'pg_config', '--includedir-server' ], '>',
+ \$stdout, '2>', \$stderr
+ or die "could not execute pg_config";
+ chomp($stdout);
+ $stdout =~ s/\r$//;
+
+ open my $header_h, '<', "$stdout/$header_path" or die "$!";
+
+ my @match = undef;
+ while (<$header_h>)
+ {
+ my $line = $_;
+
+ if (@match = $line =~ /^$regexp/)
+ {
+ last;
+ }
+ }
+
+ close $header_h;
+ die "could not find match in header $header_path\n"
+ unless @match;
+ return @match;
+}
+
+=pod
+
=item check_pg_config(regexp)
Return the number of matches of the given regular expression
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 646d6ffde4..9d8039684a 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -44,6 +44,7 @@ tests += {
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
't/038_save_logical_slots_shutdown.pl',
+ 't/039_end_of_wal.pl',
],
},
}
diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl
new file mode 100644
index 0000000000..a8b0b70bb1
--- /dev/null
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -0,0 +1,460 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+#
+# Test detecting end-of-WAL conditions. This test suite generates
+# fake defective page and record headers to trigger various failure
+# scenarios.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Fcntl qw(SEEK_SET);
+
+use integer; # causes / operator to use integer math
+
+# Header size of record header.
+my $RECORD_HEADER_SIZE = 24;
+
+# Fields retrieved from code headers.
+my @scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLOG_PAGE_MAGIC\s+(\w+)');
+my $XLP_PAGE_MAGIC = hex($scan_result[0]);
+@scan_result = scan_server_header('access/xlog_internal.h',
+ '#define\s+XLP_FIRST_IS_CONTRECORD\s+(\w+)');
+my $XLP_FIRST_IS_CONTRECORD = hex($scan_result[0]);
+
+# Values queried from the server
+my $WAL_SEGMENT_SIZE;
+my $WAL_BLOCK_SIZE;
+my $TLI;
+
+# Build path of a WAL segment.
+sub wal_segment_path
+{
+ my $node = shift;
+ my $tli = shift;
+ my $segment = shift;
+ my $wal_path =
+ sprintf("%s/pg_wal/%08X%08X%08X", $node->data_dir, $tli, 0, $segment);
+ return $wal_path;
+}
+
+# Calculate from a LSN (in bytes) its segment number and its offset.
+sub lsn_to_segment_and_offset
+{
+ my $lsn = shift;
+ return ($lsn / $WAL_SEGMENT_SIZE, $lsn % $WAL_SEGMENT_SIZE);
+}
+
+# Write some arbitrary data in WAL for the given segment at LSN.
+# This should be called while the cluster is not running.
+sub write_wal
+{
+ my $node = shift;
+ my $tli = shift;
+ my $lsn = shift;
+ my $data = shift;
+
+ my ($segment, $offset) = lsn_to_segment_and_offset($lsn);
+ my $path = wal_segment_path($node, $tli, $segment);
+
+ open my $fh, "+<:raw", $path or die;
+ seek($fh, $offset, SEEK_SET) or die;
+ print $fh $data;
+ close $fh;
+}
+
+# Emit a WAL record of arbitrary size. Returns the end LSN of the
+# record inserted, in bytes.
+sub emit_message
+{
+ my $node = shift;
+ my $size = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT pg_logical_emit_message(true, '', repeat('a', $size)) - '0/0'"
+ ));
+}
+
+# Get the current insert LSN of a node, in bytes.
+sub get_insert_lsn
+{
+ my $node = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres', "SELECT pg_current_wal_insert_lsn() - '0/0'"));
+}
+
+# Get GUC value, converted to an int.
+sub get_int_setting
+{
+ my $node = shift;
+ my $name = shift;
+ return int(
+ $node->safe_psql(
+ 'postgres',
+ "SELECT setting FROM pg_settings WHERE name = '$name'"));
+}
+
+sub start_of_page
+{
+ my $lsn = shift;
+ return $lsn & ~($WAL_BLOCK_SIZE - 1);
+}
+
+sub start_of_next_page
+{
+ my $lsn = shift;
+ return start_of_page($lsn) + $WAL_BLOCK_SIZE;
+}
+
+# Build a fake WAL record header based on the data given by the caller.
+# This needs to follow the format of the C structure XLogRecord. To
+# be inserted with write_wal().
+sub build_record_header
+{
+ my $xl_tot_len = shift;
+ my $xl_xid = shift || 0;
+ my $xl_prev = shift || 0;
+ my $xl_info = shift || 0;
+ my $xl_rmid = shift || 0;
+ my $xl_crc = shift || 0;
+
+ # This needs to follow the structure XLogRecord:
+ # I for xl_tot_len
+ # I for xl_xid
+ # Q for xl_prev
+ # C for xl_info
+ # C for xl_rmid
+ # BB for two bytes of padding
+ # I for xl_crc
+ return pack("IIQCCBBI",
+ $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+}
+
+# Build a fake WAL page header, based on the data given by the caller
+# This needs to follow the format of the C structure XLogPageHeaderData.
+# To be inserted with write_wal().
+sub build_page_header
+{
+ my $xlp_magic = shift;
+ my $xlp_info = shift || 0;
+ my $xlp_tli = shift || 0;
+ my $xlp_pageaddr = shift || 0;
+ my $xlp_rem_len = shift || 0;
+
+ # This needs to follow the structure XLogPageHeaderData:
+ # S for xlp_magic
+ # S for xlp_info
+ # I for xlp_tli
+ # Q for xlp_pageaddr
+ # I for xlp_rem_len
+ return pack("SSIQI",
+ $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+}
+
+# Make sure we are far away enough from the end of a page that we could insert
+# a couple of small records. This inserts a few records of a fixed size, until
+# the threshold gets close enough to the end of the WAL page inserting records
+# to.
+sub advance_out_of_record_splitting_zone
+{
+ my $node = shift;
+
+ my $page_threshold = $WAL_BLOCK_SIZE / 4;
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ while ($page_offset >= $WAL_BLOCK_SIZE - $page_threshold)
+ {
+ emit_message($node, $page_threshold);
+ $end_lsn = get_insert_lsn($node);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+ return $end_lsn;
+}
+
+# Advance so close to the end of a page that an XLogRecordHeader would not
+# fit on it.
+sub advance_to_record_splitting_zone
+{
+ my $node = shift;
+
+ my $end_lsn = get_insert_lsn($node);
+ my $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Get fairly close to the end of a page in big steps
+ while ($page_offset <= $WAL_BLOCK_SIZE - 512)
+ {
+ emit_message($node, $WAL_BLOCK_SIZE - $page_offset - 256);
+ $end_lsn = get_insert_lsn($node);
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+ }
+
+ # Calibrate our message size so that we can get closer 8 bytes at
+ # a time.
+ my $message_size = $WAL_BLOCK_SIZE - 80;
+ while ($page_offset <= $WAL_BLOCK_SIZE - $RECORD_HEADER_SIZE)
+ {
+ emit_message($node, $message_size);
+ $end_lsn = get_insert_lsn($node);
+
+ my $old_offset = $page_offset;
+ $page_offset = $end_lsn % $WAL_BLOCK_SIZE;
+
+ # Adjust the message size until it causes 8 bytes changes in
+ # offset, enough to be able to split a record header.
+ my $delta = $page_offset - $old_offset;
+ if ($delta > 8)
+ {
+ $message_size -= 8;
+ }
+ elsif ($delta <= 0)
+ {
+ $message_size += 8;
+ }
+ }
+ return $end_lsn;
+}
+
+# Setup a new node. The configuration chosen here minimizes the number
+# of arbitrary records that could get generated in a cluster. Enlarging
+# checkpoint_timeout avoids noise with checkpoint activity. wal_level
+# set to "minimal" avoids random standby snapshot records. Autovacuum
+# could also trigger randomly, generating random WAL activity of its own.
+my $node = PostgreSQL::Test::Cluster->new("node");
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ q[wal_level = minimal
+ autovacuum = off
+ checkpoint_timeout = '30min'
+]);
+$node->start;
+$node->safe_psql('postgres', "CREATE TABLE t AS SELECT 42");
+
+$WAL_SEGMENT_SIZE = get_int_setting($node, 'wal_segment_size');
+$WAL_BLOCK_SIZE = get_int_setting($node, 'wal_block_size');
+$TLI = $node->safe_psql('postgres',
+ "SELECT timeline_id FROM pg_control_checkpoint();");
+
+my $end_lsn;
+my $prev_lsn;
+
+###########################################################################
+note "Single-page end-of-WAL detection";
+###########################################################################
+
+# xl_tot_len is 0 (a common case, we hit trailing zeroes).
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+my $log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 0", $log_size
+ ),
+ "xl_tot_len zero");
+
+# xl_tot_len is < 24 (presumably recycled garbage).
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn, build_record_header(23));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 23",
+ $log_size),
+ "xl_tot_len short");
+
+# Need more pages, but xl_prev check fails first.
+emit_message($node, 0);
+$end_lsn = advance_out_of_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
+ "xl_prev bad");
+
+# xl_crc check fails.
+emit_message($node, 0);
+advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 10);
+$node->stop('immediate');
+# Corrupt a byte in that record, breaking its CRC.
+write_wal($node, $TLI, $end_lsn - 8, '!');
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "incorrect resource manager data checksum in record at .*", $log_size
+ ),
+ "xl_crc bad");
+
+
+###########################################################################
+note "Multi-page end-of-WAL detection, header is not split";
+###########################################################################
+
+# This series of tests requires a valid xl_prev set in the record header
+# written to WAL.
+
+# Good xl_prev, we hit zero page next (zero magic).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero");
+
+# Good xl_prev, we hit garbage page next (bad magic).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(0xcafe, 0, 1, 0));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
+ "xlp_magic bad");
+
+# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
+# xlp_pageaddr).
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
+ "xlp_info bad");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
+# continuation record.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("there is no contrecord flag at .*", $log_size),
+ "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
+
+# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
+# up.
+emit_message($node, 0);
+$prev_lsn = advance_out_of_record_splitting_zone($node);
+$end_lsn = emit_message($node, 0);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad");
+
+
+###########################################################################
+note "Multi-page, but header is split, so page checks are done first";
+###########################################################################
+
+# xl_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
+ "xlp_magic zero (split record header)");
+
+# And we'll also check xlp_pageaddr before any header checks.
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
+ "xlp_pageaddr bad (split record header)");
+
+# We'll also discover that xlp_rem_len doesn't add up before any
+# header checks,
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
+write_wal(
+ $node, $TLI,
+ start_of_next_page($end_lsn),
+ build_page_header(
+ $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
+ 1, start_of_next_page($end_lsn),
+ 123456));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid contrecord length 123456 .* at .*", $log_size),
+ "xlp_rem_len bad (split record header)");
+
+done_testing();
--
2.39.2
[application/octet-stream] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16 (25.1K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/3-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.16)
download
[application/octet-stream] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15 (24.6K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/4-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.15)
download
[application/octet-stream] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14 (22.3K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/5-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.14)
download
[application/octet-stream] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13 (22.4K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/6-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.13)
download
[application/octet-stream] v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12 (22.4K, ../../CA+hUKGKp_Go=EqA65r6naxrmXFfD-uH3DwPbm7tRCu9d2T=VyQ@mail.gmail.com/7-v8-0001-Don-t-trust-unvalidated-xl_tot_len.patch.12)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-20 00:51 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-20 00:51 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Wed, Sep 20, 2023 at 10:51:12AM +1200, Thomas Munro wrote:
> Ahh, yeah, there was a second problem screwing up the LSN accounting.
> I was trying to write correct xl_prev links, but also emitting
> optional padding records to get into the right position (depending on
> initial conditions varying between branches etc), and I forgot about
> the extra COMMIT records that emit_message() generates. Which, I
> guess, comes back to Michael's observation that this would all be a
> bit easier if we had a way to emit and flush a single record...
Unfortunately this part is not going to be backpatched based on the
current consensus, so this is goint to need an alternate and/or fluffy
solution to force the flushes at given points:
https://www.postgresql.org/message-id/20230816003333.7hn2rx5m2l7una3d@awork3.anarazel.de
The test could be largely simplified on HEAD once the flush option is
available. I'm just waiting a bit more on the other thread, keeping
an eye on the temperature, but it looks like nobody would complain to
make that optional, with the default being what we did previously, at
least..
> The solution in this version is to call get_insert_len() instead of
> using the result of emit_message() for the values returned by the
> advance_XXX() functions. The result of emit_message() is actually the
> LSN of the following COMMIT record so can't be used directly for
> building xl_prev chains.
And by doing so the test paths would enter once again in the inner
loops generating the records so as we'll only get out once we know
that we are at the border wanted, enough to stabilize the tests.
Smart.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQpCG5QanJQXNVH-@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-20 06:00 Alexander Lakhin <exclusion@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 0 replies; 86+ messages in thread
From: Alexander Lakhin @ 2023-09-20 06:00 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
20.09.2023 01:51, Thomas Munro wrote:
> The solution in this version is to call get_insert_len() instead of
> using the result of emit_message() for the values returned by the
> advance_XXX() functions. The result of emit_message() is actually the
> LSN of the following COMMIT record so can't be used directly for
> building xl_prev chains.
>
>> And may be "xlp_prev" is a typo?
> Fixed.
I've retested this version (patches for all branches, plus wal_log_hints,
plus Valgrind, and also retested code changes with my
099_restart_with_stanby.pl) and found no issues. IMHO, this patch is
ready to float off.
Thank you!
Best regards,
Alexander
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-22 03:11 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-22 03:11 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
A thought: commit 8fcb32db prevented us from logging messages that are
too big to be decoded, but it wasn't back-patched. I think that means
that in older branches, there is a behaviour change unrelated to the
"garbage bytes" problem discussed in this thread, and separate also
from the out-of-memory problem. If someone generates a record too big
to decode, say with pg_logical_emit_message(), we will fail
differently. Before this patch set, we'd bogusly detect end-of-WAL,
and after this patch we'd fail to palloc and recovery would bogusly
fail. Which outcome is more bogus is hard to answer, and clearly we
should prevent it upstream, but didn't for technical reasons. Do you
agree that that is a separate topic that doesn't prevent us from
committing this fix?
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-22 04:02 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-22 04:02 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Fri, Sep 22, 2023 at 03:11:54PM +1200, Thomas Munro wrote:
> A thought: commit 8fcb32db prevented us from logging messages that are
> too big to be decoded, but it wasn't back-patched.
Yes, there was one part done in ffd1b6bb6f8 that has required ABI
breakages, as well. There is an argument against a backpatch as a
set of records in a given range may fail to replay while they were
allowed before (I forgot the exact math, but I recall that this was
for records larger than XLogRecordMaxSize, still lower than the max
allocation mark.).
> I think that means
> that in older branches, there is a behaviour change unrelated to the
> "garbage bytes" problem discussed in this thread, and separate also
> from the out-of-memory problem. If someone generates a record too big
> to decode, say with pg_logical_emit_message(), we will fail
> differently. Before this patch set, we'd bogusly detect end-of-WAL,
> and after this patch we'd fail to palloc and recovery would bogusly
> fail. Which outcome is more bogus is hard to answer, and clearly we
> should prevent it upstream, but didn't for technical reasons. Do you
> agree that that is a separate topic that doesn't prevent us from
> committing this fix?
I don't see why it's a problem on HEAD: a startup process reacts the
same way for the end of WAL or an OOM. If we were to FATAL hard on
all stable branches for non-FRONTEND on OOM, which is something we'll
have to do anyway, then this patch set improves the situation because
we would fail in what I'd see is better in this case: with an OOM.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQ0R4kz3gN6bqS1N@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-22 23:35 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-22 23:35 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Fri, Sep 22, 2023 at 4:02 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Fri, Sep 22, 2023 at 03:11:54PM +1200, Thomas Munro wrote:
> > A thought: commit 8fcb32db prevented us from logging messages that are
> > too big to be decoded, but it wasn't back-patched.
>
> Yes, there was one part done in ffd1b6bb6f8 that has required ABI
> breakages, as well. There is an argument against a backpatch as a
> set of records in a given range may fail to replay while they were
> allowed before (I forgot the exact math, but I recall that this was
> for records larger than XLogRecordMaxSize, still lower than the max
> allocation mark.).
Hmm, OK, tricky choices. Anyway, I guess the next thing to straighten
out in this area is the OOM policy.
Pushed. Thanks Alexander and everyone for the report, testing and
help! Time to watch the build farm to see if that Perl turns out to
be robust enough...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 00:05 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 2 replies; 86+ messages in thread
From: Michael Paquier @ 2023-09-23 00:05 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 11:35:05AM +1200, Thomas Munro wrote:
> Hmm, OK, tricky choices. Anyway, I guess the next thing to straighten
> out in this area is the OOM policy.
>
> Pushed. Thanks Alexander and everyone for the report, testing and
> help! Time to watch the build farm to see if that Perl turns out to
> be robust enough...
Looks pretty stable to me. I can see one thing in wrasse related to
pack():
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2023-09-22%2023%3A23%3A23
Invalid type 'Q' in pack at t/039_end_of_wal.pl line 139.
I don't have the room to look at that now, perhaps a bit later today
depending on the flow of the wind.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQ4r1vHcryBsSi_V@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 00:16 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-23 00:16 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 12:05 PM Michael Paquier <michael@paquier.xyz> wrote:
> Looks pretty stable to me. I can see one thing in wrasse related to
> pack():
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2023-09-22%2023%3A23%3A23
> Invalid type 'Q' in pack at t/039_end_of_wal.pl line 139.
Oh, so although we were testing on 32 bit systems via CI (task
"test_world_32" on "Linux - Debian Bullseye - Meson"), in that case
the perl binary is still 64 bit. Apparently 32 bit perl binary (at
least of a certain vintage) doesn't like 'Q'.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 00:20 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-09-23 00:20 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
Thomas Munro <thomas.munro@gmail.com> writes:
> Oh, so although we were testing on 32 bit systems via CI (task
> "test_world_32" on "Linux - Debian Bullseye - Meson"), in that case
> the perl binary is still 64 bit. Apparently 32 bit perl binary (at
> least of a certain vintage) doesn't like 'Q'.
We discovered this once before, cf 945d2cb7d.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 01:07 Thomas Munro <thomas.munro@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-23 01:07 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 12:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > Oh, so although we were testing on 32 bit systems via CI (task
> > "test_world_32" on "Linux - Debian Bullseye - Meson"), in that case
> > the perl binary is still 64 bit. Apparently 32 bit perl binary (at
> > least of a certain vintage) doesn't like 'Q'.
>
> We discovered this once before, cf 945d2cb7d.
Here is my proposed fix. I couldn't quite use that trick, but we can
safely assume that the higher order bits of any LSN generated by this
test are zero. Patch tested on x86 and big-endian POWER. If no one
has a more idiomatic perl trick for this, I'll push this soon.
Attachments:
[text/x-patch] 0001-Don-t-use-Q-in-Perl-pack-strings.patch (2.3K, ../../CA+hUKGKUoRZa-YaBu+TFFCmhsKe23+7ZOb5EJRK89HraSMcLMQ@mail.gmail.com/2-0001-Don-t-use-Q-in-Perl-pack-strings.patch)
download | inline diff:
From e745858087c3130be4870c9f5fce8181736072af Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 23 Sep 2023 12:39:18 +1200
Subject: [PATCH] Don't use 'Q' in Perl pack() strings.
'Q' for 64 bit integers turns out not to work on 32 bit Perl. Use 'II'
instead, and deal with endianness.
Back-patch to 12, like bae868ca.
Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz
diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl
index a8b0b70bb1..3cc11b4b02 100644
--- a/src/test/recovery/t/039_end_of_wal.pl
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -13,6 +13,13 @@ use Fcntl qw(SEEK_SET);
use integer; # causes / operator to use integer math
+# Is this a big-endian system ("network" byte order)? We can't use 'Q' in
+# pack() calls because it's not available in some perl builds, so we need to
+# break 64 bit LSN values into two 'I' values. Fortunately we don't need to
+# deal with high values, so we can just write 0 for the high order 32 bits, but
+# we need to know the endianness to do that.
+my $BIG_ENDIAN = pack("L", 0x12345678) eq pack("N", 0x12345678);
+
# Header size of record header.
my $RECORD_HEADER_SIZE = 24;
@@ -125,13 +132,21 @@ sub build_record_header
# This needs to follow the structure XLogRecord:
# I for xl_tot_len
# I for xl_xid
- # Q for xl_prev
+ # II for xl_prev
# C for xl_info
# C for xl_rmid
# BB for two bytes of padding
# I for xl_crc
- return pack("IIQCCBBI",
- $xl_tot_len, $xl_xid, $xl_prev, $xl_info, $xl_rmid, 0, 0, $xl_crc);
+ return pack("IIIICCBBI",
+ $xl_tot_len,
+ $xl_xid,
+ $BIG_ENDIAN ? 0 : $xl_prev,
+ $BIG_ENDIAN ? $xl_prev : 0,
+ $xl_info,
+ $xl_rmid,
+ 0,
+ 0,
+ $xl_crc);
}
# Build a fake WAL page header, based on the data given by the caller
@@ -149,10 +164,15 @@ sub build_page_header
# S for xlp_magic
# S for xlp_info
# I for xlp_tli
- # Q for xlp_pageaddr
+ # II for xlp_pageaddr
# I for xlp_rem_len
- return pack("SSIQI",
- $xlp_magic, $xlp_info, $xlp_tli, $xlp_pageaddr, $xlp_rem_len);
+ return pack("SSIIII",
+ $xlp_magic,
+ $xlp_info,
+ $xlp_tli,
+ $BIG_ENDIAN ? 0 : $xlp_pageaddr,
+ $BIG_ENDIAN ? $xlp_pageaddr : 0,
+ $xlp_rem_len);
}
# Make sure we are far away enough from the end of a page that we could insert
--
2.42.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 02:02 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-23 02:02 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 12:05 PM Michael Paquier <michael@paquier.xyz> wrote:
> Looks pretty stable to me. I can see one thing in wrasse related to
> pack():
Hmm, copperhead (riscv) showed an unusual failure, a segfault in
suspiciously nearby code. I don't immediately know what that's about,
let's see if we get more clues...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 02:29 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 86+ messages in thread
From: Thomas Munro @ 2023-09-23 02:29 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 1:07 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Here is my proposed fix. I couldn't quite use that trick, but we can
> safely assume that the higher order bits of any LSN generated by this
> test are zero. Patch tested on x86 and big-endian POWER. If no one
> has a more idiomatic perl trick for this, I'll push this soon.
And done, after running through pgperltidy.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 04:44 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-23 04:44 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 02:02:02PM +1200, Thomas Munro wrote:
> Hmm, copperhead (riscv) showed an unusual failure, a segfault in
> suspiciously nearby code. I don't immediately know what that's about,
> let's see if we get more clues...
Yep, something is going on with the prefetching code:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=copperhead&dt=2023-09-22%2023%3A16%3A33
Using host libthread_db library
"/lib/riscv64-linux-gnu/libthread_db.so.1". Core was generated by
`postgres: paris: startup recovering 000000030000000000000003
'. Program terminated with signal SIGSEGV, Segmentation fault.
#0 pg_comp_crc32c_sb8 (crc=1613114916, crc@entry=4294967295,
data=data@entry=0x2af9e00d48, len=<optimized out>) at
pg_crc32c_sb8.c:56 56 uint32 a = *p4++ ^ crc;
#0 pg_comp_crc32c_sb8 (crc=1613114916, crc@entry=4294967295,
data=data@entry=0x2af9e00d48, len=<optimized out>) at
pg_crc32c_sb8.c:56
#1 0x0000002ad59a1536 in ValidXLogRecord (state=0x2af9db1fc0,
record=0x2af9e00d30, recptr=50520048) at xlogreader.c:1195
#2 0x0000002ad59a285a in XLogDecodeNextRecord
(state=state@entry=0x2af9db1fc0, nonblocking=<optimized out>) at
xlogreader.c:842
#3 0x0000002ad59a28c0 in XLogReadAhead (state=0x2af9db1fc0,
nonblocking=nonblocking@entry=false) at xlogreader.c:969
#4 0x0000002ad59a0996 in XLogPrefetcherNextBlock
(pgsr_private=184580836680, lsn=0x2af9e14618) at xlogprefetcher.c:496
#5 0x0000002ad59a11c8 in lrq_prefetch (lrq=<optimized out>) at
xlogprefetcher.c:256
#6 lrq_complete_lsn (lsn=<optimized out>, lrq=0x2af9e145c8) at
xlogprefetcher.c:294
#7 XLogPrefetcherReadRecord
(prefetcher=prefetcher@entry=0x2af9e00d48,
errmsg=errmsg@entry=0x3fec9a2bf0) at xlogprefetcher.c:1041
The stack may point out at a different issue, but perhaps this is a
matter where we're returning now XLREAD_SUCCESS where previously we
had XLREAD_FAIL, causing this code to fail thinking that the block was
valid while it's not?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQ5tIoarrcwAlbVZ@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 20:48 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-23 20:48 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sat, Sep 23, 2023 at 4:44 PM Michael Paquier <michael@paquier.xyz> wrote:
> The stack may point out at a different issue, but perhaps this is a
> matter where we're returning now XLREAD_SUCCESS where previously we
> had XLREAD_FAIL, causing this code to fail thinking that the block was
> valid while it's not?
"grison" has a little more detail -- we see
pg_comp_crc32c_sb8(len=4294636456). I'm wondering how to reproduce
this, but among the questions that jump out I have: why was it ever OK
that we load record->xl_tot_len into total_len, perform header
validation, determine that total_len < len (= this record is all on
one page, no reassembly loop needed, so now we're in the single-page
branch), then call ReadPageInternal() again, then call
ValidXLogRecord() which internally loads record->xl_tot_len *again*?
ReadPageInternal() might have changed xl_tot_len, no? That seems to
be a possible pathway to reading past the end of the buffer in the CRC
check, no?
If that value didn't change underneath us, I think we'd need an
explanation for how we finished up in the single-page branch at
xlogreader.c:842 with a large xl_tot_len, which I'm not seeing yet,
though it might take more coffee. (Possibly supporting the re-read
theory is the fact that it's only happening on a few very slow
computers, though I have no idea why it would only happen on master
[so far at least].)
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-23 23:23 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-23 23:23 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sun, Sep 24, 2023 at 09:48:42AM +1300, Thomas Munro wrote:
> "grison" has a little more detail -- we see
> pg_comp_crc32c_sb8(len=4294636456). I'm wondering how to reproduce
> this, but among the questions that jump out I have: why was it ever OK
> that we load record->xl_tot_len into total_len, perform header
> validation, determine that total_len < len (= this record is all on
> one page, no reassembly loop needed, so now we're in the single-page
> branch), then call ReadPageInternal() again, then call
> ValidXLogRecord() which internally loads record->xl_tot_len *again*?
> ReadPageInternal() might have changed xl_tot_len, no? That seems to
> be a possible pathway to reading past the end of the buffer in the CRC
> check, no?
>
> If that value didn't change underneath us, I think we'd need an
> explanation for how we finished up in the single-page branch at
> xlogreader.c:842 with a large xl_tot_len, which I'm not seeing yet,
> though it might take more coffee. (Possibly supporting the re-read
> theory is the fact that it's only happening on a few very slow
> computers, though I have no idea why it would only happen on master
> [so far at least].)
Hmm, it looks pretty clear that this is a HEAD-only thing as the
buildfarm shows and as you say, and my primary suspect here would be
71e4cc6b8ec6, I think. Any race condition underneath it would be
easier to see on slower machines. So it's likely possible that this
has messed up the page insertion logic.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZQ9zf1QO8CP4TZRO@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-24 20:02 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-24 20:02 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>
On Sun, Sep 24, 2023 at 12:23 PM Michael Paquier <michael@paquier.xyz> wrote:
> Hmm, it looks pretty clear that this is a HEAD-only thing as the
> buildfarm shows and as you say, and my primary suspect here would be
> 71e4cc6b8ec6, I think. Any race condition underneath it would be
> easier to see on slower machines. So it's likely possible that this
> has messed up the page insertion logic.
I see there was a failure on 16 on the very slow AIX box, and I have
access so looking into that...
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-24 23:58 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-24 23:58 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 09:02:35AM +1300, Thomas Munro wrote:
> I see there was a failure on 16 on the very slow AIX box, and I have
> access so looking into that...
Lucky you, if I may say ;)
A bunch of architectures that are not Intel are failing. Here is a
summary based on the buildfarm reports:
topminnow, mips64el with gcc 4.9.2
mereswine, ARMv7 with gcc 10.2.1
sungazer, ppc64 with gcc 8.3.0
frogfish, mips64el with gcc 4.6.3
mamba, macppc with gcc 10.4.0
gull, ARMv7 with clang 13.0.0
grison, ARMv7 with gcc 4.6.3
copperhead, riscv64 with gcc 10.X
The only thing close to that I have close by is tanager on Armv7 (it
has not reported to the buildfarm for a few weeks as it has
overheated because of the summer here, but I've put it back online
now). However, it has passed a few hundred cycles with both gcc and
clang yesterday, on top of having a clean buildfarm run.
With sungazer now failing on REL_16_STABLE, it feels to me that we are
actually looking at two bugs? One on HEAD, and one in stable
branches? For HEAD and the 2PC failure, the records up to PREPARE
TRANSACTION should be replayed by the standby getting promoted, but
I'd rather dig into that with a host that's able to report the
failure.
copperhead seems to be one of the failing hosts that's able to compile
things quickly. Tom, Noah, or copperhead's owner, could it be
possible to get access to one of the hosts that are failing for more
investigation? I would not do more than compiling the code and check
after the state of the 2PC test for this promotion failure.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZRDNNf8Etlvuo48a@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 00:18 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 86+ messages in thread
From: Thomas Munro @ 2023-09-25 00:18 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Sep 25, 2023 at 09:02:35AM +1300, Thomas Munro wrote:
> > I see there was a failure on 16 on the very slow AIX box, and I have
> > access so looking into that...
>
> Lucky you, if I may say ;)
FTR anyone involved with an open source project can get an account on
the GCC compile farm machines. That particular machine is so
overloaded that it's practically unusable (~8 hours to run the test,
hard to run vi etc).
> A bunch of architectures that are not Intel are failing. Here is a
> summary based on the buildfarm reports:
> topminnow, mips64el with gcc 4.9.2
> mereswine, ARMv7 with gcc 10.2.1
> sungazer, ppc64 with gcc 8.3.0
> frogfish, mips64el with gcc 4.6.3
> mamba, macppc with gcc 10.4.0
> gull, ARMv7 with clang 13.0.0
> grison, ARMv7 with gcc 4.6.3
> copperhead, riscv64 with gcc 10.X
>
> The only thing close to that I have close by is tanager on Armv7 (it
> has not reported to the buildfarm for a few weeks as it has
> overheated because of the summer here, but I've put it back online
> now). However, it has passed a few hundred cycles with both gcc and
> clang yesterday, on top of having a clean buildfarm run.
One thing that the failing systems have in common is that they are
extremely slow. 3 to 8 hours to complete the tests. turaco is an
armv7 system that doesn't fail, but it's much faster. At a guess,
probably something like an armv8 CPU that is just running 32 bit armv7
software, not a real old school armv7 chip.
Which gives me the idea to try these tests under qemu...
> With sungazer now failing on REL_16_STABLE, it feels to me that we are
> actually looking at two bugs? One on HEAD, and one in stable
> branches? For HEAD and the 2PC failure, the records up to PREPARE
> TRANSACTION should be replayed by the standby getting promoted, but
> I'd rather dig into that with a host that's able to report the
> failure.
Oh, right yeah that is quite different and could even be unrelated.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 01:32 Noah Misch <noah@leadboat.com>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 0 replies; 86+ messages in thread
From: Noah Misch @ 2023-09-25 01:32 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 01:18:56PM +1300, Thomas Munro wrote:
> On Mon, Sep 25, 2023 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
> > On Mon, Sep 25, 2023 at 09:02:35AM +1300, Thomas Munro wrote:
> > > I see there was a failure on 16 on the very slow AIX box, and I have
> > > access so looking into that...
> >
> > Lucky you, if I may say ;)
>
> FTR anyone involved with an open source project can get an account on
> the GCC compile farm machines. That particular machine is so
> overloaded that it's practically unusable (~8 hours to run the test,
> hard to run vi etc).
topminnow (gcc23) and frogfish (gcc22) are also part of the compile farm.
They're faster than sungazer (gcc111) for src/test/recovery.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 03:07 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 2 replies; 86+ messages in thread
From: Tom Lane @ 2023-09-25 03:07 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, Sep 25, 2023 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>> A bunch of architectures that are not Intel are failing. Here is a
>> summary based on the buildfarm reports:
>> mamba, macppc with gcc 10.4.0
That one is mine, so let me know if there's something particular
you'd like me to check.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 03:29 Michael Paquier <michael@paquier.xyz>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2023-09-25 03:29 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Sun, Sep 24, 2023 at 11:07:08PM -0400, Tom Lane wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
>> On Mon, Sep 25, 2023 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
>>> A bunch of architectures that are not Intel are failing. Here is a
>>> summary based on the buildfarm reports:
>>> mamba, macppc with gcc 10.4.0
>
> That one is mine, so let me know if there's something particular
> you'd like me to check.
That would help, thanks! Could it be possible to check if the
attached patch applied on HEAD influences the fate of the recovery
test 009_twophase.pl? That's one thing I am really doubting about,
and we've not done much since REL_16_STABLE has been forked..
(I'm still trying to get an environment where this can be reproduced,
but qemu takes time.)
--
Michael
Attachments:
[text/x-diff] 0001-Revert-Optimize-WAL-insertion-lock-acquisition-and-r.patch (6.3K, ../../ZRD-smlSMRiEEzEZ@paquier.xyz/2-0001-Revert-Optimize-WAL-insertion-lock-acquisition-and-r.patch)
download | inline diff:
From 3adaa49be439ba61f58f1ee8c1d2041765c247e7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 25 Sep 2023 12:24:50 +0900
Subject: [PATCH] Revert "Optimize WAL insertion lock acquisition and release
with some atomics"
This reverts commit 71e4cc6b8ec6a08f81973bd387fe575134cd0bdf.
---
src/include/storage/lwlock.h | 6 ++--
src/backend/access/transam/xlog.c | 4 +--
src/backend/storage/lmgr/lwlock.c | 46 +++++++++++++++++--------------
3 files changed, 30 insertions(+), 26 deletions(-)
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d77410bdea..34169e5889 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -129,14 +129,14 @@ extern bool LWLockAcquire(LWLock *lock, LWLockMode mode);
extern bool LWLockConditionalAcquire(LWLock *lock, LWLockMode mode);
extern bool LWLockAcquireOrWait(LWLock *lock, LWLockMode mode);
extern void LWLockRelease(LWLock *lock);
-extern void LWLockReleaseClearVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val);
+extern void LWLockReleaseClearVar(LWLock *lock, uint64 *valptr, uint64 val);
extern void LWLockReleaseAll(void);
extern bool LWLockHeldByMe(LWLock *lock);
extern bool LWLockAnyHeldByMe(LWLock *lock, int nlocks, size_t stride);
extern bool LWLockHeldByMeInMode(LWLock *lock, LWLockMode mode);
-extern bool LWLockWaitForVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval, uint64 *newval);
-extern void LWLockUpdateVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val);
+extern bool LWLockWaitForVar(LWLock *lock, uint64 *valptr, uint64 oldval, uint64 *newval);
+extern void LWLockUpdateVar(LWLock *lock, uint64 *valptr, uint64 val);
extern Size LWLockShmemSize(void);
extern void CreateLWLocks(void);
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index fcbde10529..6227230ae1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -376,7 +376,7 @@ typedef struct XLogwrtResult
typedef struct
{
LWLock lock;
- pg_atomic_uint64 insertingAt;
+ XLogRecPtr insertingAt;
XLogRecPtr lastImportantAt;
} WALInsertLock;
@@ -4634,7 +4634,7 @@ XLOGShmemInit(void)
for (i = 0; i < NUM_XLOGINSERT_LOCKS; i++)
{
LWLockInitialize(&WALInsertLocks[i].l.lock, LWTRANCHE_WAL_INSERT);
- pg_atomic_init_u64(&WALInsertLocks[i].l.insertingAt, InvalidXLogRecPtr);
+ WALInsertLocks[i].l.insertingAt = InvalidXLogRecPtr;
WALInsertLocks[i].l.lastImportantAt = InvalidXLogRecPtr;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 315a78cda9..752a5e7359 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1547,8 +1547,9 @@ LWLockAcquireOrWait(LWLock *lock, LWLockMode mode)
* *result is set to true if the lock was free, and false otherwise.
*/
static bool
-LWLockConflictsWithVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval,
- uint64 *newval, bool *result)
+LWLockConflictsWithVar(LWLock *lock,
+ uint64 *valptr, uint64 oldval, uint64 *newval,
+ bool *result)
{
bool mustwait;
uint64 value;
@@ -1572,10 +1573,13 @@ LWLockConflictsWithVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval,
*result = false;
/*
- * Reading this value atomically is safe even on platforms where uint64
- * cannot be read without observing a torn value.
+ * Read value using the lwlock's wait list lock, as we can't generally
+ * rely on atomic 64 bit reads/stores. TODO: On platforms with a way to
+ * do atomic 64 bit reads/writes the spinlock should be optimized away.
*/
- value = pg_atomic_read_u64(valptr);
+ LWLockWaitListLock(lock);
+ value = *valptr;
+ LWLockWaitListUnlock(lock);
if (value != oldval)
{
@@ -1608,8 +1612,7 @@ LWLockConflictsWithVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval,
* an implied barrier via spinlock or LWLock to avoid memory ordering issues.
*/
bool
-LWLockWaitForVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval,
- uint64 *newval)
+LWLockWaitForVar(LWLock *lock, uint64 *valptr, uint64 oldval, uint64 *newval)
{
PGPROC *proc = MyProc;
int extraWaits = 0;
@@ -1737,32 +1740,29 @@ LWLockWaitForVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 oldval,
* LWLockUpdateVar - Update a variable and wake up waiters atomically
*
* Sets *valptr to 'val', and wakes up all processes waiting for us with
- * LWLockWaitForVar(). It first sets the value atomically and then wakes up
- * waiting processes so that any process calling LWLockWaitForVar() on the same
- * lock is guaranteed to see the new value, and act accordingly.
+ * LWLockWaitForVar(). Setting the value and waking up the processes happen
+ * atomically so that any process calling LWLockWaitForVar() on the same lock
+ * is guaranteed to see the new value, and act accordingly.
*
* The caller must be holding the lock in exclusive mode.
*/
void
-LWLockUpdateVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val)
+LWLockUpdateVar(LWLock *lock, uint64 *valptr, uint64 val)
{
proclist_head wakeup;
proclist_mutable_iter iter;
PRINT_LWDEBUG("LWLockUpdateVar", lock, LW_EXCLUSIVE);
- /*
- * Note that pg_atomic_exchange_u64 is a full barrier, so we're guaranteed
- * that the variable is updated before waking up waiters.
- */
- pg_atomic_exchange_u64(valptr, val);
-
proclist_init(&wakeup);
LWLockWaitListLock(lock);
Assert(pg_atomic_read_u32(&lock->state) & LW_VAL_EXCLUSIVE);
+ /* Update the lock's value */
+ *valptr = val;
+
/*
* See if there are any LW_WAIT_UNTIL_FREE waiters that need to be woken
* up. They are always in the front of the queue.
@@ -1878,13 +1878,17 @@ LWLockRelease(LWLock *lock)
* LWLockReleaseClearVar - release a previously acquired lock, reset variable
*/
void
-LWLockReleaseClearVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val)
+LWLockReleaseClearVar(LWLock *lock, uint64 *valptr, uint64 val)
{
+ LWLockWaitListLock(lock);
+
/*
- * Note that pg_atomic_exchange_u64 is a full barrier, so we're guaranteed
- * that the variable is updated before releasing the lock.
+ * Set the variable's value before releasing the lock, that prevents race
+ * a race condition wherein a new locker acquires the lock, but hasn't yet
+ * set the variables value.
*/
- pg_atomic_exchange_u64(valptr, val);
+ *valptr = val;
+ LWLockWaitListUnlock(lock);
LWLockRelease(lock);
}
--
2.40.1
[application/pgp-signature] signature.asc (832B, ../../ZRD-smlSMRiEEzEZ@paquier.xyz/3-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 03:47 Thomas Munro <thomas.munro@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-25 03:47 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 4:07 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > On Mon, Sep 25, 2023 at 12:58 PM Michael Paquier <michael@paquier.xyz> wrote:
> >> A bunch of architectures that are not Intel are failing. Here is a
> >> summary based on the buildfarm reports:
> >> mamba, macppc with gcc 10.4.0
>
> That one is mine, so let me know if there's something particular
> you'd like me to check.
Thanks! But after spending the best part of a day installing and
compiling stuff at snail's pace, I have just this moment managed to
reproduce this reliably on an emulator running Debian armhf (= armv7).
So hopefully I can track the issue down from here without bothering
build farm owners.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 04:59 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-09-25 04:59 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, Sep 25, 2023 at 4:07 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That one is mine, so let me know if there's something particular
>> you'd like me to check.
> Thanks! But after spending the best part of a day installing and
> compiling stuff at snail's pace, I have just this moment managed to
> reproduce this reliably on an emulator running Debian armhf (= armv7).
> So hopefully I can track the issue down from here without bothering
> build farm owners.
FWIW, here's what I see on mamba:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 pg_comp_crc32c_sb8 (crc=3197562372, crc@entry=4294967295,
data=data@entry=0xfda6e030, len=<optimized out>) at pg_crc32c_sb8.c:56
56 uint32 a = *p4++ ^ crc;
(gdb) bt
#0 pg_comp_crc32c_sb8 (crc=3197562372, crc@entry=4294967295,
data=data@entry=0xfda6e030, len=<optimized out>) at pg_crc32c_sb8.c:56
#1 0x0195ae5c in ValidXLogRecord (state=0xfde5cc88, record=0xfda6e018,
recptr=<optimized out>) at xlogreader.c:1195
#2 0x0195cf18 in XLogDecodeNextRecord (state=state@entry=0xfde5cc88,
nonblocking=<optimized out>) at xlogreader.c:842
#3 0x0195d0c8 in XLogReadAhead (state=0xfde5cc88,
nonblocking=nonblocking@entry=false) at xlogreader.c:969
#4 0x01959760 in XLogPrefetcherNextBlock (pgsr_private=4259700408,
lsn=0xfda6b8a0) at xlogprefetcher.c:496
#5 0x0195a72c in lrq_prefetch (lrq=<optimized out>) at xlogprefetcher.c:256
#6 lrq_complete_lsn (lsn=50520048, lrq=0xfda6b858) at xlogprefetcher.c:294
#7 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0xfde5deb8,
errmsg=errmsg@entry=0xffffdd88) at xlogprefetcher.c:1041
#8 0x01960054 in ReadRecord (xlogprefetcher=0xfde5deb8, emode=emode@entry=15,
fetching_ckpt=fetching_ckpt@entry=false, replayTLI=replayTLI@entry=3)
at xlogrecovery.c:3067
#9 0x01963188 in PerformWalRecovery () at xlogrecovery.c:1756
#10 0x01952594 in StartupXLOG () at xlog.c:5470
#11 0x01c26e3c in StartupProcessMain () at startup.c:267
#12 0x01c1b504 in AuxiliaryProcessMain (auxtype=auxtype@entry=StartupProcess)
at auxprocess.c:141
#13 0x01c2276c in StartChildProcess (type=StartupProcess) at postmaster.c:5347
#14 0x01c260ec in PostmasterMain (argc=argc@entry=4,
argv=argv@entry=0xffffe1e4) at postmaster.c:1457
#15 0x01ee39d8 in main (argc=4, argv=0xffffe1e4) at main.c:198
(gdb) f 1
#1 0x0195ae5c in ValidXLogRecord (state=0xfde5cc88, record=0xfda6e018,
recptr=<optimized out>) at xlogreader.c:1195
1195 COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, record->xl_tot_len - SizeOfXLogRecord);
(gdb) p *record
$3 = {xl_tot_len = 0, xl_xid = 0, xl_prev = 0, xl_info = 0 '\000',
xl_rmid = 0 '\000', xl_crc = 0}
Sure looks like ValidXLogRecord is assuming that record->xl_tot_len
can be trusted without reservation.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 05:05 Thomas Munro <thomas.munro@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-25 05:05 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 5:59 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Sure looks like ValidXLogRecord is assuming that record->xl_tot_len
> can be trusted without reservation.
I think there is something at least theoretically a bit fishy about
the ancient code that re-reads the page, doesn't consider short reads,
and assumes that record->xl_tot_len hasn't changed since last time it
loaded and validated it. But I think that's also very unlikely to go
wrong, and it's not the problem here.
The problem is that bae868ca removed something we still need:
- /* XXX: more validation should be done here */
- if (total_len < SizeOfXLogRecord)
- {
- report_invalid_record(state,
-
"invalid record length at %X/%X: expected at least %u, got %u",
-
LSN_FORMAT_ARGS(RecPtr),
-
(uint32) SizeOfXLogRecord, total_len);
- goto err;
- }
+ /* We'll validate the header once we have the next page. */
gotheader = false;
If you happened to run into zeroes where an xl_tot_len is wanted right
at the end of a page (or any value not big enough to get you to the
next page), we'll fall through to the single-page branch, and then go
directly to the CRC check, but then ValidXLogRecord() subtracts
SizeOfXLogRecord and gets a crazy big length. The CRC implementation
routines on modern computers happened to use pointer arithmetic that
terminates immediately without accessing any memory, which is why
nothing was obviously wrong on most systems. The _sb8.c
implementation for older ARM, MIPS etc use a length-based loop, and
read off into deep space.
Draft patch attached, including a new test for 039_end_of_wal.pl that
fails on all systems without the above code.
Attachments:
[text/x-patch] 0001-Fix-edge-case-for-xl_tot_len-missed-by-bae868ca.patch (2.5K, ../../CA+hUKGLCkTT7zYjzOxuLGahBdQ=McF=z5ZvrjSOnW4EDhVjT-g@mail.gmail.com/2-0001-Fix-edge-case-for-xl_tot_len-missed-by-bae868ca.patch)
download | inline diff:
From 6bb759dc22839f8ec101cad8551738be71b9ea9d Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 25 Sep 2023 17:19:23 +1300
Subject: [PATCH] Fix edge-case for xl_tot_len missed by bae868ca.
bae868ca removed a check that was still needed. If you had a value
in xl_tot_len position that was too small for a record header, but not
big enough to span onto the next page, we'd immediately perform the CRC
check using a very large length, due to modulo arithmetic. Because of
arbitrary coding differences between the CRC implementations on
different platforms, nothing bad would happen on common modern systems.
On systems using the _sb8.c fallback, we could segfault.
Fix, and add a new test for that case. Without the code change, the
test would fail because recovery would report a CRC failure on typical
systems (instead of "too small"), but on some systems it would segfault.
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 58654b746c..0bba56cc0e 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -653,6 +653,15 @@ restart:
}
else
{
+ /* There may be no next page if it's too small. */
+ if (total_len < SizeOfXLogRecord)
+ {
+ report_invalid_record(state,
+ "invalid record length at %X/%X: expected at least %u, got %u",
+ LSN_FORMAT_ARGS(RecPtr),
+ (uint32) SizeOfXLogRecord, total_len);
+ goto err;
+ }
/* We'll validate the header once we have the next page. */
gotheader = false;
}
diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl
index 61728bc38b..a523c16af7 100644
--- a/src/test/recovery/t/039_end_of_wal.pl
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -281,6 +281,20 @@ ok( $node->log_contains(
$log_size),
"xl_tot_len short");
+# xl_tot_len in final position, not big enough to span into a new page but
+# also not eligible for regular record header validation
+emit_message($node, 0);
+$end_lsn = advance_to_record_splitting_zone($node);
+$node->stop('immediate');
+write_wal($node, $TLI, $end_lsn,
+ build_record_header(1, 0, 0xdeadbeef));
+$log_size = -s $node->logfile;
+$node->start;
+ok( $node->log_contains(
+ "invalid record length at .*: expected at least 24, got 1", $log_size
+ ),
+ "xl_tot_len too small at end-of-page");
+
# Need more pages, but xl_prev check fails first.
emit_message($node, 0);
$end_lsn = advance_out_of_record_splitting_zone($node);
--
2.39.2
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 05:13 Tom Lane <tgl@sss.pgh.pa.us>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-09-25 05:13 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
Michael Paquier <michael@paquier.xyz> writes:
> On Sun, Sep 24, 2023 at 11:07:08PM -0400, Tom Lane wrote:
>> That one is mine, so let me know if there's something particular
>> you'd like me to check.
> That would help, thanks! Could it be possible to check if the
> attached patch applied on HEAD influences the fate of the recovery
> test 009_twophase.pl? That's one thing I am really doubting about,
> and we've not done much since REL_16_STABLE has been forked..
Applying this does *not* fix it. Stack trace looks the same
as in my previous message.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 05:29 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-09-25 05:29 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
Thomas Munro <thomas.munro@gmail.com> writes:
> If you happened to run into zeroes where an xl_tot_len is wanted right
> at the end of a page (or any value not big enough to get you to the
> next page), we'll fall through to the single-page branch, and then go
> directly to the CRC check, but then ValidXLogRecord() subtracts
> SizeOfXLogRecord and gets a crazy big length. The CRC implementation
> routines on modern computers happened to use pointer arithmetic that
> terminates immediately without accessing any memory, which is why
> nothing was obviously wrong on most systems. The _sb8.c
> implementation for older ARM, MIPS etc use a length-based loop, and
> read off into deep space.
Ah-hah. Maybe there should be an Assert in the modern CRC code path
that detects the bogus arguments?
> Draft patch attached, including a new test for 039_end_of_wal.pl that
> fails on all systems without the above code.
Both the 009 and 039 tests pass on mamba with this patch. I did
not read the code, but testing-wise it seems good.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 06:52 Michael Paquier <michael@paquier.xyz>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2023-09-25 06:52 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 01:13:37AM -0400, Tom Lane wrote:
> Applying this does *not* fix it. Stack trace looks the same
> as in my previous message.
Okay. Thanks anyway for the test!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZREuHPvLt-LvZGIv@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-25 22:14 Thomas Munro <thomas.munro@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 86+ messages in thread
From: Thomas Munro @ 2023-09-25 22:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Mon, Sep 25, 2023 at 6:29 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Ah-hah. Maybe there should be an Assert in the modern CRC code path
> that detects the bogus arguments?
Yeah. Assertion added, and pushed. Thanks for confirming on mamba,
and thanks also to Alexander for an off-list test report.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: BUG #17928: Standby fails to decode WAL on termination of primary
@ 2023-09-26 00:34 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2023-09-26 00:34 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Alexander Lakhin <exclusion@gmail.com>; Sergei Kornilov <sk@zsrv.org>; Noah Misch <noah@leadboat.com>; Kyotaro Horiguchi <horikyota.ntt@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>; pgbf@twiska.com
On Tue, Sep 26, 2023 at 11:14:52AM +1300, Thomas Munro wrote:
> Yeah. Assertion added, and pushed. Thanks for confirming on mamba,
> and thanks also to Alexander for an off-list test report.
Yeah++, thanks!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZRInMNv3ByxkEo8l@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
end of thread, other threads:[~2023-09-26 00:34 UTC | newest]
Thread overview: 86+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-05-11 08:00 BUG #17928: Standby fails to decode WAL on termination of primary PG Bug reporting form <noreply@postgresql.org>
2023-05-11 18:00 ` Alexander Lakhin <exclusion@gmail.com>
2023-05-15 03:38 ` Thomas Munro <thomas.munro@gmail.com>
2023-07-10 20:00 ` Noah Misch <noah@leadboat.com>
2023-07-11 06:54 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-07-17 00:49 ` Noah Misch <noah@leadboat.com>
2023-07-20 06:01 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2023-08-10 07:45 ` Michael Paquier <michael@paquier.xyz>
2023-08-10 23:22 ` Michael Paquier <michael@paquier.xyz>
2023-08-11 02:58 ` Noah Misch <noah@leadboat.com>
2023-08-11 06:08 ` Michael Paquier <michael@paquier.xyz>
2023-08-11 14:00 ` Noah Misch <noah@leadboat.com>
2023-08-11 23:56 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-12 00:20 ` Michael Paquier <michael@paquier.xyz>
2023-08-12 00:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-12 21:13 ` Noah Misch <noah@leadboat.com>
2023-08-13 03:12 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-13 03:30 ` Noah Misch <noah@leadboat.com>
2023-08-14 02:08 ` Michael Paquier <michael@paquier.xyz>
2023-08-14 03:56 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-14 10:28 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-15 02:04 ` Michael Paquier <michael@paquier.xyz>
2023-08-15 02:36 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-15 03:00 ` Michael Paquier <michael@paquier.xyz>
2023-08-15 06:11 ` Michael Paquier <michael@paquier.xyz>
2023-08-16 03:29 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-16 06:17 ` Michael Paquier <michael@paquier.xyz>
2023-08-18 00:20 ` Michael Paquier <michael@paquier.xyz>
2023-08-18 02:30 ` Thomas Munro <thomas.munro@gmail.com>
2023-08-18 04:55 ` Michael Paquier <michael@paquier.xyz>
2023-08-20 23:32 ` Michael Paquier <michael@paquier.xyz>
2023-09-02 01:29 ` Michael Paquier <michael@paquier.xyz>
2023-09-02 11:51 ` Sergei Kornilov <sk@zsrv.org>
2023-09-02 23:42 ` Michael Paquier <michael@paquier.xyz>
2023-09-03 05:43 ` Michael Paquier <michael@paquier.xyz>
2023-09-03 11:21 ` Sergei Kornilov <sk@zsrv.org>
2023-09-04 03:20 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-04 03:54 ` Michael Paquier <michael@paquier.xyz>
2023-09-04 07:17 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-04 07:28 ` Michael Paquier <michael@paquier.xyz>
2023-09-04 07:43 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-05 03:19 ` Michael Paquier <michael@paquier.xyz>
2023-09-05 03:58 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-05 04:13 ` Michael Paquier <michael@paquier.xyz>
2023-09-12 01:01 ` Michael Paquier <michael@paquier.xyz>
2023-09-12 02:08 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-12 02:12 ` Michael Paquier <michael@paquier.xyz>
2023-09-15 22:07 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-16 00:02 ` Michael Paquier <michael@paquier.xyz>
2023-09-16 00:20 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-16 04:13 ` Michael Paquier <michael@paquier.xyz>
2023-09-16 06:00 ` Alexander Lakhin <exclusion@gmail.com>
2023-09-17 20:46 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-19 09:25 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-19 09:53 ` Michael Paquier <michael@paquier.xyz>
2023-09-19 10:09 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-19 13:00 ` Alexander Lakhin <exclusion@gmail.com>
2023-09-19 22:51 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-20 00:51 ` Michael Paquier <michael@paquier.xyz>
2023-09-22 03:11 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-22 04:02 ` Michael Paquier <michael@paquier.xyz>
2023-09-22 23:35 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 00:05 ` Michael Paquier <michael@paquier.xyz>
2023-09-23 00:16 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 00:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-09-23 01:07 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 02:29 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 02:02 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 04:44 ` Michael Paquier <michael@paquier.xyz>
2023-09-23 20:48 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-23 23:23 ` Michael Paquier <michael@paquier.xyz>
2023-09-24 20:02 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-24 23:58 ` Michael Paquier <michael@paquier.xyz>
2023-09-25 00:18 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-25 01:32 ` Noah Misch <noah@leadboat.com>
2023-09-25 03:07 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-09-25 03:29 ` Michael Paquier <michael@paquier.xyz>
2023-09-25 05:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-09-25 06:52 ` Michael Paquier <michael@paquier.xyz>
2023-09-25 03:47 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-25 04:59 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-09-25 05:05 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-25 05:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-09-25 22:14 ` Thomas Munro <thomas.munro@gmail.com>
2023-09-26 00:34 ` Michael Paquier <michael@paquier.xyz>
2023-09-20 06:00 ` Alexander Lakhin <exclusion@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox