From: Zhijie Hou <houzhijie22@gmail.com>
To: Melanie Plageman <melanieplageman@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Rogers Wang <rogers.ww@qq.com>
Cc: pgsql-hackers@lists.postgresql.org, Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Álvaro Herrera <alvherre@kurilemu.de>
Cc: Andres Freund <andres@anarazel.de>
Subject: Re: WAL_LOG CREATE DATABASE strategy broken for non-standard page layouts
Date: Tue, 17 Nov 2026 23:20:54 +0800
Message-ID: <2889116c-7da1-4ab0-b10a-3fadb99c24e7@gmail.com> (raw)
In-Reply-To: <CAAKRu_YAW5i_T65T38BwDhMuXT9x2Aw+PjKXqyLYXa409i-BNg@mail.gmail.com>
References: <202602021525.qv4m7oge4eca@alvherre.pgsql>
<tencent_2E870046716FD94285045E96505A2D4E2908@qq.com>
<apUTKUUvYfG3SlBF@paquier.xyz>
<CAAKRu_YE-T_qObGxn5gC-b27umwCMGKP40eaOKhVAbTJ9rJhXQ@mail.gmail.com>
<CA+TgmobaT5h77ZUheD7uoDcK+vRj63Eaa9q8muo8RtqdfctLog@mail.gmail.com>
<CAAKRu_ZxxVLPfpZiX4L+QURW6HO60v8mFSRn64zado4kzygt5w@mail.gmail.com>
<CAAKRu_ZK6ihUvzFL8Uie2QNdGSbhAD8TemUPoURxsXk48ospgA@mail.gmail.com>
<CA+TgmobaoRn6AEEv19Ri1BbMyUS0aOMNdcAOzShDn2JW8sq21w@mail.gmail.com>
<CAAKRu_b9r-Pz9gzYvC+5EHnXqNgfsJhcH=huUGbXzvcnbHtxCg@mail.gmail.com>
<6e62d593-a8b2-415d-834d-e54ed519bb04@gmail.com>
<CAAKRu_YAW5i_T65T38BwDhMuXT9x2Aw+PjKXqyLYXa409i-BNg@mail.gmail.com>
Hi,
On Tue, Nov17, 2026 at 01:35 AM Melanie Plageman wrote:
> On Sat, Nov 14, 2026 at 8:08 AM Zhijie Hou <houzhijie22@gmail.com> wrote:
> >
> > I just have one minor comment on the TAP test in 0001. IIUC, the
> > pre-existing test in 034_create_database.pl was added by commit
> > 8a8661828ae to verify that when CREATE DATABASE loads the template's
> > pg_class into shared_buffers, it marks those buffers as permanent so
> > that a subsequent checkpoint flushes them to disk.
> >
> > But with 0001, the modified test vacuuming tab_db_before_create_1 in
> > template1 beforehand loads template1's pg_class into shared_buffers
> > with BM_PERMANENT already set before CREATE DATABASE runs. If we
> > want to keep the original regression coverage, maybe we could run
> > the new test steps after the pre-existing test.
>
> Ah, thanks so much for catching this. I read the test and not the
> commit message adding it and thought it was just trying to make sure
> that tables added after the copy didn't end up in the copied database
> -- which, if I think about it, would have been a silly test. I've
> changed it in attached v3 to retain the coverage in the initial test.
>
> I also realized while editing this that I'll have to make a wrapper
> version of CreateAndCopyRelationData() for the backports since it is
> public. A bit annoying but I think better than the alternatives.
Thanks for updating the patch.
I think we also need to increase shared_buffers (e.g., to 16MB), because
$node->init(allows_streaming => 1) sets shared_buffers to 1MB
internally, causing the pg_class pages to be evicted from shared_buffers
before CREATE TABLE runs, so the test could not reach the old bug path.
BTW, while looking into why the wal_consistency_checking test didn't
catch this bug, I noticed that verifyBackupPageConsistency() also masks
out all the bits in VM pages via heap_mask() (somewhat similar to this
bug, since it treats pd_lower..pd_upper as unused space), so the
consistency check for VM pages always "passes". Once we avoid masking
the VM bitmap there (see the attached draft patch for reference), the
existing test catches this CREATE DB WAL_LOG bug right away:
PG_TEST_EXTRA=wal_consistency_checking meson test -C build
recovery/027_stream_regress
So if it's worth increasing the possibility of catching similar VM bugs,
maybe we could improve that as well (in a separate patch or thread).
--
Best Regards,
Zhijie Hou
Attachments:nocfbot-0001-Fix-WAL-consistency-checking-for-visibility-map-p.patch (text/plain, 2.6K) download
From 4ede72e8ee6d79cda8209a725b1b6b04e07438a4 Mon Sep 17 00:00:00 2001
From: Zhijie Hou <houzhijie22@gmail.com>
Date: Tue, 17 Nov 2026 22:06:38 +0800
Subject: [PATCH v1 1/2] Fix WAL consistency checking for visibility map pages.
Both RM_HEAP_ID and RM_HEAP2_ID WAL records register VISIBILITYMAP_FORKNUM
buffers (since a507b86900f for VM-setting records, and since ed62d26caca
for VM-clearing records). When wal_consistency_checking is enabled for
heap or heap2, verifyBackupPageConsistency() previously passed every
registered block to rmgr.rm_mask (heap_mask()) without checking forknum.
Because a visibility map page is initialized with PageInit(page, BLCKSZ, 0)
and stores its bitmap between pd_lower (SizeOfPageHeaderData) and pd_upper
(BLCKSZ), mask_unused_space() in heap_mask() zeroed out the entire bitmap
in both the WAL backup image and the replay image before comparing them.
Consequently, wal_consistency_checking silently missed visibility map page
divergences during WAL replay.
Fix verifyBackupPageConsistency() to mask only the page LSN and checksum
when forknum is VISIBILITYMAP_FORKNUM, avoiding an ABI break to
RmgrData.rm_mask.
---
src/backend/access/transam/xlogrecovery.c | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.cindex 52ac6d58423..490e51f26e5 100644--- a/src/backend/access/transam/xlogrecovery.c+++ b/src/backend/access/transam/xlogrecovery.c@@ -30,6 +30,7 @@
#include <sys/time.h>
#include <unistd.h>
+#include "access/bufmask.h"
#include "access/timeline.h"
#include "access/transam.h"
#include "access/xact.h"
@@ -2547,10 +2548,21 @@ verifyBackupPageConsistency(XLogReaderState *record)
errmsg_internal("%s", record->errormsg_buf)));
/*
- * If masking function is defined, mask both the primary and replay- * images+ * Mask both the primary and replay images before comparing them.+ *+ * Visibility map pages are modified by RM_HEAP_ID and RM_HEAP2_ID+ * records, but do not follow the standard heap page layout: their+ * bitmap occupies the bytes between pd_lower and pd_upper (see+ * visibilitymap.c), which heap_mask() would wipe via+ * mask_unused_space(). Only the LSN and checksum can legitimately+ * differ on a visibility map page.
*/
- if (rmgr.rm_mask != NULL)+ if (forknum == VISIBILITYMAP_FORKNUM)+ {+ mask_page_lsn_and_checksum(replay_image_masked);+ mask_page_lsn_and_checksum(primary_image_masked);+ }+ else if (rmgr.rm_mask != NULL)
{
rmgr.rm_mask(replay_image_masked, blkno);
rmgr.rm_mask(primary_image_masked, blkno);
--
2.54.0 (Apple Git-157)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-hackers@postgresql.org
Cc: houzhijie22@gmail.com, melanieplageman@gmail.com, robertmhaas@gmail.com, michael@paquier.xyz, rogers.ww@qq.com, boekewurm+postgres@gmail.com, alvherre@kurilemu.de, andres@anarazel.de
Subject: Re: WAL_LOG CREATE DATABASE strategy broken for non-standard page layouts
In-Reply-To: <2889116c-7da1-4ab0-b10a-3fadb99c24e7@gmail.com>
References: <202602021525.qv4m7oge4eca@alvherre.pgsql> <tencent_2E870046716FD94285045E96505A2D4E2908@qq.com> <apUTKUUvYfG3SlBF@paquier.xyz> <CAAKRu_YE-T_qObGxn5gC-b27umwCMGKP40eaOKhVAbTJ9rJhXQ@mail.gmail.com> <CA+TgmobaT5h77ZUheD7uoDcK+vRj63Eaa9q8muo8RtqdfctLog@mail.gmail.com> <CAAKRu_ZxxVLPfpZiX4L+QURW6HO60v8mFSRn64zado4kzygt5w@mail.gmail.com> <CAAKRu_ZK6ihUvzFL8Uie2QNdGSbhAD8TemUPoURxsXk48ospgA@mail.gmail.com> <CA+TgmobaoRn6AEEv19Ri1BbMyUS0aOMNdcAOzShDn2JW8sq21w@mail.gmail.com> <CAAKRu_b9r-Pz9gzYvC+5EHnXqNgfsJhcH=huUGbXzvcnbHtxCg@mail.gmail.com> <6e62d593-a8b2-415d-834d-e54ed519bb04@gmail.com> <CAAKRu_YAW5i_T65T38BwDhMuXT9x2Aw+PjKXqyLYXa409i-BNg@mail.gmail.com> <2889116c-7da1-4ab0-b10a-3fadb99c24e7@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox