pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix possibility of logical decoding partial transaction changes. 7+ messages / 1 participants [nested] [flat]
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/1b370758770c3259f2ec09eefe47274be995792e Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 +++- src/backend/replication/logical/snapbuild.c | 19 ++++++--- src/include/replication/logical.h | 6 +++ 6 files changed, 118 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/cf2c69ec5a5b8ac437c85b8a21d02dc011a78886 Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 +++- src/backend/replication/logical/snapbuild.c | 19 ++++++--- src/include/replication/logical.h | 6 +++ 6 files changed, 118 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f7d3caf9ddbfee2567d834ec4639ef717240809f Modified Files -------------- contrib/test_decoding/Makefile | 2 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 +++- src/backend/replication/logical/snapbuild.c | 19 ++++++--- src/include/replication/logical.h | 6 +++ 6 files changed, 117 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/aee8c2b95492e1b0a228184cd9147b1c0749d673 Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 +++- src/backend/replication/logical/snapbuild.c | 19 ++++++--- src/include/replication/logical.h | 6 +++ 6 files changed, 118 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/2f3304ce133f4e9c0992ff72ece47093833da55d Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ contrib/test_decoding/meson.build | 1 + .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 +++- src/backend/replication/logical/snapbuild.c | 19 ++++++--- src/include/replication/logical.h | 6 +++ 7 files changed, 119 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/068674f4ab44f2c891b6841432687958e8e9c9a6 Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ contrib/test_decoding/meson.build | 1 + .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 ++-- src/backend/replication/logical/snapbuild.c | 29 +++++++++++--- src/include/replication/snapbuild.h | 1 + 7 files changed, 122 insertions(+), 10 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix possibility of logical decoding partial transaction changes. @ 2024-07-11 13:49 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-07-11 13:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix possibility of logical decoding partial transaction changes. When creating and initializing a logical slot, the restart_lsn is set to the latest WAL insertion point (or the latest replay point on standbys). Subsequently, WAL records are decoded from that point to find the start point for extracting changes in the DecodingContextFindStartpoint() function. Since the initial restart_lsn could be in the middle of a transaction, the start point must be a consistent point where we won't see the data for partial transactions. Previously, when not building a full snapshot, serialized snapshots were restored, and the SnapBuild jumps to the consistent state even while finding the start point. Consequently, the slot's restart_lsn and confirmed_flush could be set to the middle of a transaction. This could lead to various unexpected consequences. Specifically, there were reports of logical decoding decoding partial transactions, and assertion failures occurred because only subtransactions were decoded without decoding their top-level transaction until decoding the commit record. To resolve this issue, the changes prevent restoring the serialized snapshot and jumping to the consistent state while finding the start point. On v17 and HEAD, a flag indicating whether snapshot restores should be skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION has been bumpded. On backbranches, the flag is stored in the LogicalDecodingContext instead, preserving on-disk compatibility. Backpatch to all supported versions. Reported-by: Drew Callahan Reviewed-by: Amit Kapila, Hayato Kuroda Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com Backpatch-through: 12 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/bb19b70081e2248f242cd00227abff5b1e105eb6 Modified Files -------------- contrib/test_decoding/Makefile | 3 +- .../expected/skip_snapshot_restore.out | 45 +++++++++++++++++++++ contrib/test_decoding/meson.build | 1 + .../test_decoding/specs/skip_snapshot_restore.spec | 46 ++++++++++++++++++++++ src/backend/replication/logical/logical.c | 7 ++-- src/backend/replication/logical/snapbuild.c | 29 +++++++++++--- src/include/replication/snapbuild.h | 1 + 7 files changed, 122 insertions(+), 10 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-07-11 13:49 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org> 2024-07-11 13:49 pgsql: Fix possibility of logical decoding partial transaction changes. Masahiko Sawada <msawada@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox