pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix corruption when relation truncation fails. 6+ messages / 1 participants [nested] [flat]
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:00 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:00 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/38c579b08988e6f1a5bd74241d0a1001421d8015 Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 29 ++++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 89 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:00 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:00 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/0350b876b074dc307b82ba18cd3c7cad46066baf Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 28 +++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 88 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:00 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:00 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/ba02d24bacbb221891370a6b2c2655f24f128305 Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 29 ++++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 89 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:00 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:00 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/fb540b6aa5abe68d7ba1d84a90f5afd9c78564e3 Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 29 ++++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 89 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:00 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:00 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/23c743b645a546f7ec7aae0a3d7201328d6994b3 Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 32 ++++++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 92 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix corruption when relation truncation fails. @ 2024-12-20 11:01 Thomas Munro <tmunro@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Thomas Munro @ 2024-12-20 11:01 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption when relation truncation fails. RelationTruncate() does three things, while holding an AccessExclusiveLock and preventing checkpoints: 1. Logs the truncation. 2. Drops buffers, even if they're dirty. 3. Truncates some number of files. Step 2 could previously be canceled if it had to wait for I/O, and step 3 could and still can fail in file APIs. All orderings of these operations have data corruption hazards if interrupted, so we can't give up until the whole operation is done. When dirty pages were discarded but the corresponding blocks were left on disk due to ERROR, old page versions could come back from disk, reviving deleted data (see pgsql-bugs #18146 and several like it). When primary and standby were allowed to disagree on relation size, standbys could panic (see pgsql-bugs #18426) or revive data unknown to visibility management on the primary (theorized). Changes: * WAL is now unconditionally flushed first * smgrtruncate() is now called in a critical section, preventing interrupts and causing PANIC on file API failure * smgrtruncate() has a new parameter for existing fork sizes, because it can't call smgrnblocks() itself inside a critical section The changes apply to RelationTruncate(), smgr_redo() and pg_truncate_visibility_map(). That last is also brought up to date with other evolutions of the truncation protocol. The VACUUM FileTruncate() failure mode had been discussed in older reports than the ones referenced below, with independent analysis from many people, but earlier theories on how to fix it were too complicated to back-patch. The more recently invented cancellation bug was diagnosed by Alexander Lakhin. Other corruption scenarios were spotted by me while iterating on this patch and earlier commit 75818b3a. Back-patch to all supported releases. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reported-by: rootcause000@gmail.com Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/18146-04e908c662113ad5%40postgresql.org Discussion: https://postgr.es/m/18426-2d18da6586f152d6%40postgresql.org Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/2280912165d62a8b1de477818a405a76ffc66b2e Modified Files -------------- contrib/pg_visibility/pg_visibility.c | 32 ++++++++++++++++++++----- src/backend/catalog/storage.c | 44 +++++++++++++++++++++++++---------- src/backend/storage/smgr/md.c | 28 +++++++++++++++------- src/backend/storage/smgr/smgr.c | 14 +++++++---- src/include/storage/md.h | 2 +- src/include/storage/smgr.h | 5 ++-- 6 files changed, 92 insertions(+), 33 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2024-12-20 11:01 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-12-20 11:00 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org> 2024-12-20 11:00 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org> 2024-12-20 11:00 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org> 2024-12-20 11:00 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org> 2024-12-20 11:00 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org> 2024-12-20 11:01 pgsql: Fix corruption when relation truncation fails. Thomas Munro <tmunro@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox