agora inbox for pgsql-committers@postgresql.orghelp / color / mirror / Atom feed
pgsql: Fix VM clear WAL logging by registering VM blocks 4+ messages / 1 participants [nested] [flat]
* pgsql: Fix VM clear WAL logging by registering VM blocks @ 2026-07-15 21:25 Melanie Plageman <melanieplageman@gmail.com> 0 siblings, 0 replies; 4+ messages in thread From: Melanie Plageman @ 2026-07-15 21:25 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix VM clear WAL logging by registering VM blocks Heap WAL records that clear bits on the visibility map (like inserts and deletes) did not register the visibility map blocks they modified. Because the WAL summarizer only records registered blocks, an incremental backup taken over such operations would omit the changed VM pages. On restore, the VM would retain stale all-visible/all-frozen bits, which can cause wrong results from index-only scans and incorrect relfrozenxid advancement due to vacuum page skipping. Not registering the VM buffer also meant we never emitted FPIs of VM pages when clearing bits. A torn VM page won't raise an error because the VM is read with ZERO_ON_ERROR; with checksums on, it would be detected and zeroed, but with checksums off, it is accepted as-is and can lead to data corruption. Fix this by registering the VM buffer in the WAL record when clearing VM bits. The VM buffer must now be locked throughout the critical section that modifies the VM and heap pages and emits the WAL record. This can slow down operations that clear the VM, since the VM lock is held longer and VM FPIs may be emitted, but it is required for correctness. Note that this fix does not repair existing incremental backups. Bumps XLOG_PAGE_MAGIC. Author: Melanie Plageman <melanieplageman@gmail.com> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/flat/CAAKRu_bn%2Be7F4yPFBgFbnP%2BsyJRKyNK092bjD2LKvZW7O4Svag; Backpatch-through: 17 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/ed62d26cacac96ca5c6b4e6fcc5308b11a8371c8 Modified Files -------------- contrib/pg_surgery/heap_surgery.c | 40 +++- src/backend/access/heap/heapam.c | 370 +++++++++++++++++++++++++++----- src/backend/access/heap/heapam_xlog.c | 192 +++++++++++------ src/backend/access/heap/pruneheap.c | 2 + src/backend/access/heap/visibilitymap.c | 28 +-- src/bin/pg_walsummary/t/002_blocks.pl | 7 +- src/include/access/heapam_xlog.h | 17 +- src/include/access/xlog_internal.h | 2 +- 8 files changed, 500 insertions(+), 158 deletions(-) ^ permalink raw reply [nested|flat] 4+ messages in thread
* pgsql: Fix VM clear WAL logging by registering VM blocks @ 2026-07-15 21:32 Melanie Plageman <melanieplageman@gmail.com> 0 siblings, 0 replies; 4+ messages in thread From: Melanie Plageman @ 2026-07-15 21:32 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix VM clear WAL logging by registering VM blocks Heap WAL records that clear bits on the visibility map (like inserts and deletes) did not register the visibility map blocks they modified. Because the WAL summarizer only records registered blocks, an incremental backup taken over such operations would omit the changed VM pages. On restore, the VM would retain stale all-visible/all-frozen bits, which can cause wrong results from index-only scans and incorrect relfrozenxid advancement due to vacuum page skipping. Not registering the VM buffer also meant we never emitted FPIs of VM pages when clearing bits. A torn VM page won't raise an error because the VM is read with ZERO_ON_ERROR; with checksums on, it would be detected and zeroed, but with checksums off, it is accepted as-is and can lead to data corruption. Fix this by registering the VM buffer in the WAL record when clearing VM bits. The VM buffer must now be locked throughout the critical section that modifies the VM and heap pages and emits the WAL record. This can slow down operations that clear the VM, since the VM lock is held longer and VM FPIs may be emitted, but it is required for correctness. Note that this fix does not repair existing incremental backups. Bumps XLOG_PAGE_MAGIC. Though it is late in the cycle (post-beta 2) to be doing so, that seemed better than maintaining the backwards compatability code in yet another branch. Author: Melanie Plageman <melanieplageman@gmail.com> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/flat/CAAKRu_bn%2Be7F4yPFBgFbnP%2BsyJRKyNK092bjD2LKvZW7O4Svag; Backpatch-through: 17 Branch ------ REL_19_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/b01c31eef9c3a83d0bd9f30656cedaa6722890ee Modified Files -------------- contrib/pg_surgery/heap_surgery.c | 40 +++- src/backend/access/heap/heapam.c | 371 ++++++++++++++++++++++++++------ src/backend/access/heap/heapam_xlog.c | 202 +++++++++++------ src/backend/access/heap/pruneheap.c | 2 + src/backend/access/heap/visibilitymap.c | 24 ++- src/bin/pg_walsummary/t/002_blocks.pl | 7 +- src/include/access/heapam_xlog.h | 17 +- src/include/access/xlog_internal.h | 2 +- 8 files changed, 508 insertions(+), 157 deletions(-) ^ permalink raw reply [nested|flat] 4+ messages in thread
* pgsql: Fix VM clear WAL logging by registering VM blocks @ 2026-07-15 21:38 Melanie Plageman <melanieplageman@gmail.com> 0 siblings, 0 replies; 4+ messages in thread From: Melanie Plageman @ 2026-07-15 21:38 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix VM clear WAL logging by registering VM blocks Heap WAL records that clear bits on the visibility map (like inserts and deletes) did not register the visibility map blocks they modified. Because the WAL summarizer only records registered blocks, an incremental backup taken over such operations would omit the changed VM pages. On restore, the VM would retain stale all-visible/all-frozen bits, which can cause wrong results from index-only scans and incorrect relfrozenxid advancement due to vacuum page skipping. Not registering the VM buffer also meant we never emitted FPIs of VM pages when clearing bits. A torn VM page won't raise an error because the VM is read with ZERO_ON_ERROR; with checksums on, it would be detected and zeroed, but with checksums off, it is accepted as-is and can lead to data corruption. Fix this by registering the VM buffer in the WAL record when clearing VM bits. The VM buffer must now be locked throughout the critical section that modifies the VM and heap pages and emits the WAL record. This can slow down operations that clear the VM, since the VM lock is held longer and VM FPIs may be emitted, but it is required for correctness. Note that this fix does not repair existing incremental backups. Author: Melanie Plageman <melanieplageman@gmail.com> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/flat/CAAKRu_bn%2Be7F4yPFBgFbnP%2BsyJRKyNK092bjD2LKvZW7O4Svag; Backpatch-through: 17 Branch ------ REL_18_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f581fa729d8e108fef853c3156267b1f753d0210 Modified Files -------------- contrib/pg_surgery/heap_surgery.c | 40 +++- src/backend/access/heap/heapam.c | 397 ++++++++++++++++++++++++++------ src/backend/access/heap/heapam_xlog.c | 252 ++++++++++++++------ src/backend/access/heap/visibilitymap.c | 34 ++- src/bin/pg_walsummary/t/002_blocks.pl | 7 +- src/include/access/heapam_xlog.h | 18 +- src/include/access/visibilitymap.h | 2 + 7 files changed, 595 insertions(+), 155 deletions(-) ^ permalink raw reply [nested|flat] 4+ messages in thread
* pgsql: Fix VM clear WAL logging by registering VM blocks @ 2026-07-15 21:44 Melanie Plageman <melanieplageman@gmail.com> 0 siblings, 0 replies; 4+ messages in thread From: Melanie Plageman @ 2026-07-15 21:44 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix VM clear WAL logging by registering VM blocks Heap WAL records that clear bits on the visibility map (like inserts and deletes) did not register the visibility map blocks they modified. Because the WAL summarizer only records registered blocks, an incremental backup taken over such operations would omit the changed VM pages. On restore, the VM would retain stale all-visible/all-frozen bits, which can cause wrong results from index-only scans and incorrect relfrozenxid advancement due to vacuum page skipping. Not registering the VM buffer also meant we never emitted FPIs of VM pages when clearing bits. A torn VM page won't raise an error because the VM is read with ZERO_ON_ERROR; with checksums on, it would be detected and zeroed, but with checksums off, it is accepted as-is and can lead to data corruption. Fix this by registering the VM buffer in the WAL record when clearing VM bits. The VM buffer must now be locked throughout the critical section that modifies the VM and heap pages and emits the WAL record. This can slow down operations that clear the VM, since the VM lock is held longer and VM FPIs may be emitted, but it is required for correctness. Note that this fix does not repair existing incremental backups. Author: Melanie Plageman <melanieplageman@gmail.com> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/flat/CAAKRu_bn%2Be7F4yPFBgFbnP%2BsyJRKyNK092bjD2LKvZW7O4Svag; Backpatch-through: 17 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/c0d9864f5ce47902dd258bb61dd89ada6d020b55 Modified Files -------------- contrib/pg_surgery/heap_surgery.c | 40 +- src/backend/access/heap/heapam.c | 650 +++++++++++++++++++++++++------- src/backend/access/heap/visibilitymap.c | 34 +- src/bin/pg_walsummary/t/002_blocks.pl | 7 +- src/include/access/heapam_xlog.h | 18 +- src/include/access/visibilitymap.h | 2 + 6 files changed, 596 insertions(+), 155 deletions(-) ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-07-15 21:44 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-07-15 21:25 pgsql: Fix VM clear WAL logging by registering VM blocks Melanie Plageman <melanieplageman@gmail.com> 2026-07-15 21:32 pgsql: Fix VM clear WAL logging by registering VM blocks Melanie Plageman <melanieplageman@gmail.com> 2026-07-15 21:38 pgsql: Fix VM clear WAL logging by registering VM blocks Melanie Plageman <melanieplageman@gmail.com> 2026-07-15 21:44 pgsql: Fix VM clear WAL logging by registering VM blocks Melanie Plageman <melanieplageman@gmail.com>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox