pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi 5+ messages / 1 participants [nested] [flat]
* pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi @ 2023-03-08 05:55 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 5+ messages in thread From: Andres Freund @ 2023-03-08 05:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids When vacuum_defer_cleanup_age is bigger than the current xid, including the epoch, the subtraction of vacuum_defer_cleanup_age would lead to a wrapped around xid. While that normally is not a problem, the subsequent conversion to a 64bit xid results in a 64bit-xid very far into the future. As that xid is used as a horizon to detect whether rows versions are old enough to be removed, that allows removal of rows that are still visible (i.e. corruption). If vacuum_defer_cleanup_age was never changed from the default, there is no chance of this bug occurring. This bug was introduced in dc7420c2c92. A lesser version of it exists in 12-13, introduced by fb5344c969a, affecting only GiST. The 12-13 version of the issue can, in rare cases, lead to pages in a gist index getting recycled too early, potentially causing index entries to be found multiple times. The fix is fairly simple - don't allow vacuum_defer_cleanup_age to retreat further than FirstNormalTransactionId. Patches to make similar bugs easier to find, by adding asserts to the 64bit xid infrastructure, have been proposed, but are not suitable for backpatching. Currently there are no tests for vacuum_defer_cleanup_age. A patch introducing infrastructure to make writing a test easier has been posted to the list. Reported-by: Michail Nikolaev <michail.nikolaev@gmail.com> Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20230108002923.cyoser3ttmt63bfn@awork3.anarazel.de Backpatch: 12-, but impact/fix is smaller for 12-13 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/be504a3e974d75be6f95c8f9b7367126034f2d12 Modified Files -------------- src/backend/storage/ipc/procarray.c | 85 +++++++++++++++++++++++++++++++------ 1 file changed, 73 insertions(+), 12 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi @ 2023-03-08 05:55 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 5+ messages in thread From: Andres Freund @ 2023-03-08 05:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids When vacuum_defer_cleanup_age is bigger than the current xid, including the epoch, the subtraction of vacuum_defer_cleanup_age would lead to a wrapped around xid. While that normally is not a problem, the subsequent conversion to a 64bit xid results in a 64bit-xid very far into the future. As that xid is used as a horizon to detect whether rows versions are old enough to be removed, that allows removal of rows that are still visible (i.e. corruption). If vacuum_defer_cleanup_age was never changed from the default, there is no chance of this bug occurring. This bug was introduced in dc7420c2c92. A lesser version of it exists in 12-13, introduced by fb5344c969a, affecting only GiST. The 12-13 version of the issue can, in rare cases, lead to pages in a gist index getting recycled too early, potentially causing index entries to be found multiple times. The fix is fairly simple - don't allow vacuum_defer_cleanup_age to retreat further than FirstNormalTransactionId. Patches to make similar bugs easier to find, by adding asserts to the 64bit xid infrastructure, have been proposed, but are not suitable for backpatching. Currently there are no tests for vacuum_defer_cleanup_age. A patch introducing infrastructure to make writing a test easier has been posted to the list. Reported-by: Michail Nikolaev <michail.nikolaev@gmail.com> Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20230108002923.cyoser3ttmt63bfn@awork3.anarazel.de Backpatch: 12-, but impact/fix is smaller for 12-13 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/391f08fd68047881345a69e5a7fff173d8f9c897 Modified Files -------------- src/backend/storage/ipc/procarray.c | 85 +++++++++++++++++++++++++++++++------ 1 file changed, 73 insertions(+), 12 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi @ 2023-03-08 05:55 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 5+ messages in thread From: Andres Freund @ 2023-03-08 05:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids When vacuum_defer_cleanup_age is bigger than the current xid, including the epoch, the subtraction of vacuum_defer_cleanup_age would lead to a wrapped around xid. While that normally is not a problem, the subsequent conversion to a 64bit xid results in a 64bit-xid very far into the future. As that xid is used as a horizon to detect whether rows versions are old enough to be removed, that allows removal of rows that are still visible (i.e. corruption). If vacuum_defer_cleanup_age was never changed from the default, there is no chance of this bug occurring. This bug was introduced in dc7420c2c92. A lesser version of it exists in 12-13, introduced by fb5344c969a, affecting only GiST. The 12-13 version of the issue can, in rare cases, lead to pages in a gist index getting recycled too early, potentially causing index entries to be found multiple times. The fix is fairly simple - don't allow vacuum_defer_cleanup_age to retreat further than FirstNormalTransactionId. Patches to make similar bugs easier to find, by adding asserts to the 64bit xid infrastructure, have been proposed, but are not suitable for backpatching. Currently there are no tests for vacuum_defer_cleanup_age. A patch introducing infrastructure to make writing a test easier has been posted to the list. Reported-by: Michail Nikolaev <michail.nikolaev@gmail.com> Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20230108002923.cyoser3ttmt63bfn@awork3.anarazel.de Backpatch: 12-, but impact/fix is smaller for 12-13 Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/324281fd5b1b413b70910b8c59d00900efedd256 Modified Files -------------- src/backend/storage/ipc/procarray.c | 85 +++++++++++++++++++++++++++++++------ 1 file changed, 73 insertions(+), 12 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi @ 2023-03-08 05:55 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 5+ messages in thread From: Andres Freund @ 2023-03-08 05:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids When vacuum_defer_cleanup_age is bigger than the current xid, including the epoch, the subtraction of vacuum_defer_cleanup_age would lead to a wrapped around xid. While that normally is not a problem, the subsequent conversion to a 64bit xid results in a 64bit-xid very far into the future. As that xid is used as a horizon to detect whether rows versions are old enough to be removed, that allows removal of rows that are still visible (i.e. corruption). If vacuum_defer_cleanup_age was never changed from the default, there is no chance of this bug occurring. This bug was introduced in dc7420c2c92. A lesser version of it exists in 12-13, introduced by fb5344c969a, affecting only GiST. The 12-13 version of the issue can, in rare cases, lead to pages in a gist index getting recycled too early, potentially causing index entries to be found multiple times. The fix is fairly simple - don't allow vacuum_defer_cleanup_age to retreat further than FirstNormalTransactionId. Patches to make similar bugs easier to find, by adding asserts to the 64bit xid infrastructure, have been proposed, but are not suitable for backpatching. Currently there are no tests for vacuum_defer_cleanup_age. A patch introducing infrastructure to make writing a test easier has been posted to the list. Reported-by: Michail Nikolaev <michail.nikolaev@gmail.com> Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20230108002923.cyoser3ttmt63bfn@awork3.anarazel.de Backpatch: 12-, but impact/fix is smaller for 12-13 Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/e6d77f22c7a8bed88ceef3a3267931f4ce988664 Modified Files -------------- src/backend/utils/time/snapmgr.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi @ 2023-03-08 05:55 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 5+ messages in thread From: Andres Freund @ 2023-03-08 05:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids When vacuum_defer_cleanup_age is bigger than the current xid, including the epoch, the subtraction of vacuum_defer_cleanup_age would lead to a wrapped around xid. While that normally is not a problem, the subsequent conversion to a 64bit xid results in a 64bit-xid very far into the future. As that xid is used as a horizon to detect whether rows versions are old enough to be removed, that allows removal of rows that are still visible (i.e. corruption). If vacuum_defer_cleanup_age was never changed from the default, there is no chance of this bug occurring. This bug was introduced in dc7420c2c92. A lesser version of it exists in 12-13, introduced by fb5344c969a, affecting only GiST. The 12-13 version of the issue can, in rare cases, lead to pages in a gist index getting recycled too early, potentially causing index entries to be found multiple times. The fix is fairly simple - don't allow vacuum_defer_cleanup_age to retreat further than FirstNormalTransactionId. Patches to make similar bugs easier to find, by adding asserts to the 64bit xid infrastructure, have been proposed, but are not suitable for backpatching. Currently there are no tests for vacuum_defer_cleanup_age. A patch introducing infrastructure to make writing a test easier has been posted to the list. Reported-by: Michail Nikolaev <michail.nikolaev@gmail.com> Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20230108002923.cyoser3ttmt63bfn@awork3.anarazel.de Backpatch: 12-, but impact/fix is smaller for 12-13 Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/3c92f7e9d851084bf51cb9231353c54474fe0438 Modified Files -------------- src/backend/utils/time/snapmgr.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2023-03-08 05:55 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-03-08 05:55 pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi Andres Freund <andres@anarazel.de> 2023-03-08 05:55 pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi Andres Freund <andres@anarazel.de> 2023-03-08 05:55 pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi Andres Freund <andres@anarazel.de> 2023-03-08 05:55 pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi Andres Freund <andres@anarazel.de> 2023-03-08 05:55 pgsql: Fix corruption due to vacuum_defer_cleanup_age underflowing 64bi Andres Freund <andres@anarazel.de>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox