pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
7+ messages / 1 participants
[nested] [flat]

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this change adds the list of transaction IDs and
sub-transaction IDs, that have modified catalogs and are running during
snapshot serialization, to the serialized snapshot. After restart or
otherwise, when we restore from such a serialized snapshot, the
corresponding list is restored in memory. Now, when decoding a COMMIT
record, we check both the list and the ReorderBuffer to see if the
transaction has modified catalogs.

Since this adds additional information to the serialized snapshot, we
cannot backpatch it. For back branches, we took another approach.
We remember the last-running-xacts list of the decoded RUNNING_XACTS
record after restoring the previously serialized snapshot. Then, we mark
the transaction as containing catalog changes if it's in the list of
initial running transactions and its commit record has
XACT_XINFO_HAS_INVALS. This doesn't require any file format changes but
the transaction will end up being added to the snapshot even if it has
only relcache invalidations. But that won't be a problem since we use
snapshot built during decoding only to read system catalogs.

This commit bumps SNAPBUILD_VERSION because of a change in SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/7f13ac812313666a2fbb8dacfbee67e78d2ba0bc

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 ++++
.../specs/catalog_change_snapshot.spec             |  39 +++
src/backend/replication/logical/decode.c           |   3 +-
src/backend/replication/logical/reorderbuffer.c    |  71 +++++-
src/backend/replication/logical/snapbuild.c        | 273 ++++++++++++++-------
src/include/replication/reorderbuffer.h            |  12 +
src/include/replication/snapbuild.h                |   2 +-
8 files changed, 353 insertions(+), 93 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_15_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/272248a0c1b18a82c4266e0dc3b526d4d2637de3

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  15 +++
src/backend/replication/logical/snapbuild.c        | 137 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 232 insertions(+), 8 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/68dcce247f1a13318613a0e27782b2ca21a4ceb7

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  15 +++
src/backend/replication/logical/snapbuild.c        | 137 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 232 insertions(+), 8 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/547b963683e34e9f26401093131f9bea89d48968

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  15 ++-
src/backend/replication/logical/snapbuild.c        | 133 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 227 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/794460783125c4588bd426eac3d6508b1e45dc07

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  15 ++-
src/backend/replication/logical/snapbuild.c        | 133 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 227 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/e721123b7a0ca1a33d5315d81b32751936a0aa6c

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  44 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  15 ++-
src/backend/replication/logical/snapbuild.c        | 135 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 229 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix catalog lookup with the wrong snapshot during logical decodi
@ 2022-08-11 04:52 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-08-11 04:52 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix catalog lookup with the wrong snapshot during logical decoding.

Previously, we relied on HEAP2_NEW_CID records and XACT_INVALIDATION
records to know if the transaction has modified the catalog, and that
information is not serialized to snapshot. Therefore, after the restart,
if the logical decoding decodes only the commit record of the transaction
that has actually modified a catalog, we will miss adding its XID to the
snapshot. Thus, we will end up looking at catalogs with the wrong
snapshot.

To fix this problem, this changes the snapshot builder so that it
remembers the last-running-xacts list of the decoded RUNNING_XACTS record
after restoring the previously serialized snapshot. Then, we mark the
transaction as containing catalog changes if it's in the list of initial
running transactions and its commit record has XACT_XINFO_HAS_INVALS. To
avoid ABI breakage, we store the array of the initial running transactions
in the static variables InitialRunningXacts and NInitialRunningXacts,
instead of storing those in SnapBuild or ReorderBuffer.

This approach has a false positive; we could end up adding the transaction
that didn't change catalog to the snapshot since we cannot distinguish
whether the transaction has catalog changes only by checking the COMMIT
record. It doesn't have the information on which (sub) transaction has
catalog changes, and XACT_XINFO_HAS_INVALS doesn't necessarily indicate
that the transaction has catalog change. But that won't be a problem since
we use snapshot built during decoding only to read system catalogs.

On the master branch, we took a more future-proof approach by writing
catalog modifying transactions to the serialized snapshot which avoids the
above false positive. But we cannot backpatch it because of a change in
the SnapBuild.

Reported-by: Mike Oh
Author: Masahiko Sawada
Reviewed-by: Amit Kapila, Shi yu, Takamichi Osumi, Kyotaro Horiguchi, Bertrand Drouvot, Ahsan Hadi
Backpatch-through: 10
Discussion: https://postgr.es/m/81D0D8B0-E7C4-4999-B616-1E5004DBDCD2%40amazon.com

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/bf0718c137c4d2c297b1b0ae2bd1d0ae7055940e

Modified Files
--------------
contrib/test_decoding/Makefile                     |   2 +-
.../expected/catalog_change_snapshot.out           |  41 +++++++
.../specs/catalog_change_snapshot.spec             |  39 ++++++
src/backend/replication/logical/decode.c           |  16 ++-
src/backend/replication/logical/snapbuild.c        | 135 +++++++++++++++++++--
src/include/replication/snapbuild.h                |   3 +
6 files changed, 227 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2022-08-11 04:52 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>
2022-08-11 04:52 pgsql: Fix catalog lookup with the wrong snapshot during logical decodi Amit Kapila <akapila@postgresql.org>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox