pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix catalog lookup with the wrong snapshot during logical decodi 7+ messages / 1 participants [nested] [flat]
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/272248a0c1b18a82c4266e0dc3b526d4d2637de3 Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 15 +++ src/backend/replication/logical/snapbuild.c | 137 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 232 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this change adds the list of transaction IDs and sub-transaction IDs, that have modified catalogs and are running during snapshot serialization, to the serialized snapshot. After restart or otherwise, when we restore from such a serialized snapshot, the corresponding list is restored in memory. Now, when decoding a COMMIT record, we check both the list and the ReorderBuffer to see if the transaction has modified catalogs. Since this adds additional information to the serialized snapshot, we cannot backpatch it. For back branches, we took another approach. We remember the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. This doesn't require any file format changes but the transaction will end up being added to the snapshot even if it has only relcache invalidations. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. This commit bumps SNAPBUILD_VERSION because of a change in SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/7f13ac812313666a2fbb8dacfbee67e78d2ba0bc Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 ++++ .../specs/catalog_change_snapshot.spec | 39 +++ src/backend/replication/logical/decode.c | 3 +- src/backend/replication/logical/reorderbuffer.c | 71 +++++- src/backend/replication/logical/snapbuild.c | 273 ++++++++++++++------- src/include/replication/reorderbuffer.h | 12 + src/include/replication/snapbuild.h | 2 +- 8 files changed, 353 insertions(+), 93 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/e721123b7a0ca1a33d5315d81b32751936a0aa6c Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 15 ++- src/backend/replication/logical/snapbuild.c | 135 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 229 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/547b963683e34e9f26401093131f9bea89d48968 Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 15 ++- src/backend/replication/logical/snapbuild.c | 133 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 227 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_10_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/bf0718c137c4d2c297b1b0ae2bd1d0ae7055940e Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 41 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 16 ++- src/backend/replication/logical/snapbuild.c | 135 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 227 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/68dcce247f1a13318613a0e27782b2ca21a4ceb7 Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 15 +++ src/backend/replication/logical/snapbuild.c | 137 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 232 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi @ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix catalog lookup with the wrong snapshot during logical decoding. Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION records to know if the transaction has modified the catalog, and that information is not serialized to snapshot. Therefore, after the restart, if the logical decoding decodes only the commit record of the transaction that has actually modified a catalog, we will miss adding its XID to the snapshot. Thus, we will end up looking at catalogs with the wrong snapshot. To fix this problem, this changes the snapshot builder so that it remembers the last-running-xacts list of the decoded RUNNING_XACTS record after restoring the previously serialized snapshot. Then, we mark the transaction as containing catalog changes if it's in the list of initial running transactions and its commit record has XACT_XINFO_HAS_INVALS. To avoid ABI breakage, we store the array of the initial running transactions in the static variables InitialRunningXacts and NInitialRunningXacts, instead of storing those in SnapBuild or ReorderBuffer. This approach has a false positive; we could end up adding the transaction that didn't change catalog to the snapshot since we cannot distinguish whether the transaction has catalog changes only by checking the COMMIT record. It doesn't have the information on which (sub) transaction has catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate that the transaction has catalog change. But that won't be a problem since we use snapshot built during decoding only to read system catalogs. On the master branch, we took a more future-proof approach by writing catalog modifying transactions to the serialized snapshot which avoids the above false positive. But we cannot backpatch it because of a change in the SnapBuild. Reported-by: Mike Oh Author: Masahiko Sawada Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi Backpatch-through: 10 Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/794460783125c4588bd426eac3d6508b1e45dc07 Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/catalog_change_snapshot.out | 44 +++++++ .../specs/catalog_change_snapshot.spec | 39 ++++++ src/backend/replication/logical/decode.c | 15 ++- src/backend/replication/logical/snapbuild.c | 133 +++++++++++++++++++-- src/include/replication/snapbuild.h | 3 + 6 files changed, 227 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2022-08-11 04:52 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org> 2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox